Skip to main content

[javaee-spec users] Re: How can serious TCK issues be addressed?

  • From: Bill Shannon <bill.shannon@...>
  • To: arjan tijms <arjan.tijms@...>
  • Cc: users <users@...>
  • Subject: [javaee-spec users] Re: How can serious TCK issues be addressed?
  • Date: Mon, 21 Oct 2013 17:50:43 -0700

arjan tijms wrote on 10/21/13 5:09 PM:
> Hi,
>
>
> On Tue, Oct 22, 2013 at 1:11 AM, Bill Shannon <bill.shannon@...
> <mailto:bill.shannon@...>> wrote:
>
>>
>>     Would something like this possible? Asking the community to submit 
>> known
>>     failure cases and then testing that a new version of the TCK indeed 
>> fails
>>     on those?
>     Well, sort of.  [...]
>
>     Since new substantial tests of this sort would typically only be added 
> to
>     a new version of the TCK corresponding to a new version of the spec, all
>     existing products are going to fail that version of the TCK until 
> they're
>     updated to the new version of the spec.
>
>
> Indeed, so that's maybe a bit of the issue here.
>
> A TCK is made final before or simultaneously with the new version of the 
> spec,
> isn't it? But there aren't any real products other than the RI then to test 
> it
> against.
Simultaneously with.

> So while most likely every existing product implementing Java EE x is going 
> to
> fail the full TCK for x + 1, what I'd hoped is that it would be possible to
> just run a single test in isolation and test that this test indeed fails on
> the existing product for Java EE x.
You can certainly look at the pass/fail details for individual tests.

> For instance, in Java EE 6 not all implementations call a JASPIC auth module
> for a non-protected resource, while the spec says they should. For Java EE 
> 7 I
> would have liked to be able to ask that the implementation where I know this
> doesn't happen is tested for this particular test. I know that the existing
> implementation could theoretically never pass the whole test suite since 
> some
> requirements have been added for Java EE 7, but I would be interested only 
> in
> an "auth module called for public page" test, and the assurance that it 
> indeed
> failed on the specific implementation that I mentioned. 
>
> That isolated test would not be Java EE 7/JASPIC 1.1 specific, since nothing
> specific changed in that area between versions. A test that just requests a
> non-protected resource, then checks if an auth module has indeed been 
> invoked
> makes just as much sense on a Java EE 6 implementation as it does on a Java 
> EE
> 7 one. The only difference would be that the test just wasn't in the Java 
> EE 6
> TCK.
>
> I guess the TCK currently only tests if new products comply? If indeed so, I
> think it might not be a bad idea to also test known (old) failure cases to 
> see
> if the TCK indeed fails correctly when it needs to fail.
What you're asking would require a lot more coordination than we currently 
do. 
Remember that we don't test all the products; each vendor tests their own 
product.

>  
>
>     And through all these discussions about what *could* happen, you have to
>     realize that our TCK development resources are limited and we have to
>     prioritize which tests to spend time creating so not every test that you
>     can think of will be added to the TCK.
>
>
> Here it would be beneficial if people could contribute tests I guess, but I
> understand this is not as relatively trivial as say contributing a patch to
> Mojarra or so.
Licensees have always been able to contribute tests.  It essentially never 
happens.

>  
>
>     I don't remember if that's still in the TCK license or not; you can read
>     it as well as I can if you really care.
>
>
> Okay, I'll try to find the license then and see if it's still in there.
The licenses are linked from the JSR pages on jcp.org.



[javaee-spec users] How can serious TCK issues be addressed?

arjan tijms 10/20/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

Mark Struberg 10/21/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

arjan tijms 10/21/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

Bill Shannon 10/21/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

Bill Shannon 10/21/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

Mark Struberg 10/21/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

Bill Shannon 10/21/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

arjan tijms 10/21/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

Bill Shannon 10/21/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

arjan tijms 10/22/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

Bill Shannon 10/22/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

Pete Muir 10/22/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

Edward Burns 10/23/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

arjan tijms 10/23/2013

[javaee-spec users] Re: Re: How can serious TCK issues be addressed?

Pete Muir 10/23/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

arjan tijms 10/24/2013

[javaee-spec users] Re: How can serious TCK issues be addressed?

Bill Shannon 10/21/2013
 
 
Close
loading
Please Confirm
Close