According to the JAXP documentation, http://jaxp.java.net/1.4/JAXP-Compatibility.html#JAXP_security, is it no longer possible to use XSLT extension functions when a security manager is set. This is a major regression added by JAXP in commit #2679. This limitation does not come from Xerces and the Xerces team seems to agree that it is not a good idea.
This new and unavoidable behaviour breaks all the applications using a security manager (hello RMI) with no possible workaround. Setting a security manager does not means that the application will parse user provided XML/XSLT files. It should be up to the application to (un)set the secure mode. A method to disable the secure mode even when a security manager is set should be provided.