Issue Details (XML | Word | Printable)

Key: GLASSFISH-18327
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Blocker Blocker
Assignee: Byron Nevins
Reporter: jp2011
Votes: 0
Watchers: 3
Operations

If you were logged in you would be able to see more operations.
glassfish

install-node-dcom does not abide by --windowsdomain parameter

Created: 06/Feb/12 06:51 AM   Updated: 04/Oct/13 08:48 PM   Resolved: 06/Feb/12 11:47 PM
Component/s: distributed management
Affects Version/s: 3.1.2_b20
Fix Version/s: 3.1.2_b21, 3.1.2.2

Time Tracking:
Not Specified

File Attachments: 1. Zip Archive 18327.zip (5 kB) 06/Feb/12 10:05 PM - Byron Nevins

Environment:

Windows 2008 R2
Glassfish 3.1.2_b20


Tags: 3_1_2-approved
Participants: Byron Nevins, easarina, jp2011, mtobler and rdblaha1


 Description  « Hide

Running the command install-node-dcom ignores the --windowsdomain or -d parameters.

asadmin> install-node-dcom --windowsdomain domain.net
Enter the value for the hosts operand> somehost
Enter remote password for serviceaccount@somehost>
com.sun.enterprise.util.cluster.windows.process.WindowsException: Logon failure:
unknown user name or bad password.
Command install-node-dcom failed.

As can be seen, the windows domain is ignored and the asadmin is attempting to connect to somehost with a local account.



Byron Nevins added a comment - 06/Feb/12 06:19 PM

The command obviously fell through the cracks and received no testing.

Confirmed problem.


Byron Nevins added a comment - 06/Feb/12 09:15 PM
  • What is the impact on the customer of the bug?
    Bad. He will not be able to use a Windows Domain Controller for authenticated login to remote machines in the domain.
    Whether a real user would really ever do this is debatable. I would not. I would create local accounts on each machine.

How likely is it that a customer will see the bug and how serious is the bug?
100% chance if they run the command with the windows domain option

Is it a regression?
No.
Does it meet other bug fix criteria (security, performance, etc.)?
Yes

  • What is the cost of fixing the bug?
    1 man-day.

How risky is the fix?
Little or no risk. I will thoroughly test that it continues to work both before & after the fix with NON-DOMAIN authentication.
Unfortunately I don't think I will be able to test domain controller authentication because:
(a) My company refuses to give me multiple Windows computers. Therefore I can't create my own domain.
(b) The computer in a domain that I can access is provisioned and it is locked down too tightly to access with SAMBA/jcifs.

How much work is the fix?
See "cost" above.

Is the fix complicated?
No. Very straight-forward.

  • Is there an impact on documentation or message strings?
    No.
  • Which tests should QA (re)run to verify the fix did not destabilize GlassFish?
    Can't possibly destabilize since it is a local-only command.
    QA should run normal tests and should try it against a domain-controller authenticated machine.
  • Which is the targeted build of 3.1.2 for this fix?
    The last one.

Byron Nevins added a comment - 06/Feb/12 10:05 PM

Bugfix - source and diffs


Byron Nevins added a comment - 06/Feb/12 10:35 PM

I tested for regressions thoroughly – I installed on a remote in my LAN. I also installed over VPN to a lab machine.

This guarantees that the existing behavior still works fine for non Domain Controller authentication.

It SHOULD work for Domain Controller authentication but I can't test it.
There's no risk since before the fix – Domain Controller authentication definitely did not work. It can't possibly be any worse and it probably is fixed now.

Worst case scenario is that a user would have to run the command against a local account on the machine or he would need to install GF manually on the remote machine(s).


Byron Nevins added a comment - 06/Feb/12 11:47 PM

Sending cli\src\main\java\com\sun\enterprise\admin\cli\cluster\InstallNodeDcomCommand.java
Transmitting file data .
Committed revision 52465.


jp2011 added a comment - 09/Feb/12 02:19 AM

Sorry to tell you but this isn't fixed in bundle 21.

Performing a network capture while running the command still shows the host name being used instead of the domain for passing the credentials. I'm not sure how you are validating the domain in the OK method, but this may not be correct and hence why the host is still being used instead of the domain.

I'm also fairly certain the other dcom commands are not working with the windowsdomain switch either.


easarina added a comment - 06/Mar/12 07:36 PM

I believe that this is a regression issue and the error message was seen because was not use -w <user_name> option. See, for example:
asadmin install-node-dcom -W password1.txt bigapp-oblade-3
com.sun.enterprise.util.cluster.windows.process.WindowsException: Logon failure:
unknown user name or bad password.
Command install-node-dcom failed.

But the command: "asadmin install-node-dcom -w aroot -W password1.txt bigapp-oblade-3" was executed successfully. On the DAS machine and on the remote host was used the same user aroot, and according to the help: "The default is the user that is running this subcommand.". I.e. in my case aroot.


jp2011 added a comment - 06/Mar/12 08:07 PM

The command that I used was correct I just didn't spell it out in the notes. Trying the command as you listed also doesn't work and specifies a local windows account on the remote host. I have also tried a local windows account and the command doesn't work. The user is successfully authenticated but errors still occur with the SMB side (see GLASSFISH-18451).

Furthermore, the GUI behaves the same as my attempt to run the commands manually from the command line.


easarina added a comment - 06/Mar/12 08:42 PM

Does validate-dcom command work?


jp2011 added a comment - 06/Mar/12 09:06 PM

Password file contains the following line: AS_ADMIN_WINDOWSPASSWORD=${ALIAS=glassfish-alias}

I have setup the alias already in asadmin as per the documentation.

c:\glassfish3\bin>asadmin --passwordfile passwordfile.txt validate-dcom -w glassfish remotehost
remote failure:
Successfully verified that the host, remotehost, is not the local machine as required.
Successfully resolved host name to: remotehost/10.65.30.187
Successfully connected to DCOM Port at port 135 on host remotehost.
Successfully connected to NetBIOS Session Service at port 139 on host remotehost
nc.
Successfully connected to Windows Shares at port 445 on host remotehost.
The remote file, C: doesn't exist on remotehost: Access is denied.

Command validate-dcom failed.


Byron Nevins added a comment - 06/Mar/12 09:21 PM

Bug located – The NtlmPasswordAuthentication object which does the actual Windows authentication – was ignoring the domain!

==========================================================================

Index: src/main/java/com/sun/enterprise/util/cluster/windows/io/WindowsRemoteFileSystem.java
===================================================================
— src/main/java/com/sun/enterprise/util/cluster/windows/io/WindowsRemoteFileSystem.java (revision 52702)
+++ src/main/java/com/sun/enterprise/util/cluster/windows/io/WindowsRemoteFileSystem.java (working copy)
@@ -37,7 +37,6 @@

  • only if the new code is made subject to such option by the copyright
  • holder.
    */
    -
    package com.sun.enterprise.util.cluster.windows.io;

import com.sun.enterprise.util.cluster.windows.process.WindowsCredentials;
@@ -55,9 +54,25 @@
private final NtlmPasswordAuthentication authorization;

public WindowsRemoteFileSystem(WindowsCredentials cr) { - host = getIP(cr.getHost()); - authorization = new NtlmPasswordAuthentication(host, cr.getUser(), cr.getPassword()); + // if host and domain are the same we can use the IP address of the host + // otherwise use the domain name. + boolean useDomain; + String hostName = cr.getHost(); + String domain = cr.getDomain(); + + if(!domain.equals(hostName)) + useDomain=true; + else + useDomain = false; + + host = getIP(hostName); + + if(useDomain) + authorization = new NtlmPasswordAuthentication(domain, cr.getUser(), cr.getPassword()); + else + authorization = new NtlmPasswordAuthentication(host, cr.getUser(), cr.getPassword()); }
+
public WindowsRemoteFileSystem(String hostname, NtlmPasswordAuthentication auth) {
host = getIP(hostname);
authorization = auth;
@@ -85,7 +100,8 @@
private String getIP(String hostname) {
try { return InetAddress.getByName(hostname).getHostAddress(); - } catch (Exception e) { + }
+ catch (Exception e) { return hostname; }
}


easarina added a comment - 06/Mar/12 09:53 PM

For me install-node-dcom failed, if -w <user_name> was not used.


easarina added a comment - 06/Mar/12 11:56 PM

If validate-dcom doesn't work (Access is denied), then DCOM was not enabled correctly (or was not enabled at all). Please see how to enable dcom in the 3.1.2 High Availability Administration Guide. In my environment, both validate-dcom and install-node-dcom work.

asadmin validate-dcom -W password1.txt bigapp-oblade-3
Command validate-dcom executed successfully.

I will open a minor regression bug regarding: install-node-dcom doesn't work without -w <user_name>.


jp2011 added a comment - 07/Mar/12 01:54 PM

I appreciate your thought that I haven't enabled DCOM correctly and I would love it if that was the case but so far there is nothing that has been suggested or mentioned that I haven't already done. I see other people on the internet having the same issue without an answer and would like to help get to the bottom of the issue.

If this is a configuration issue in windows, it isn't currently documented in the 3.1.2 documents. If this is a code issue that I have run into that you haven't, it needs to be identified and fixed for the good of the community. Could you please email me offline and we can better investigate the cause and possible resolution?

Many thanks.


easarina added a comment - 07/Mar/12 05:39 PM

There is no problems in a code, if DCOM works install-node-dcom and other commands work. But DCOM has to be enabled. The 3.1.2 High Availability Administration Guide has a detail instruction how to do it. You need to edit the Windows registry manually. Did you see and execute DCOM enabling procedure?


jp2011 added a comment - 07/Mar/12 06:01 PM - edited

In fact there are issues in code with the DCOM validation and missing Windows configuration steps in the documentation. Byron and I have been discussing this yesterday and he has found another bug when using domain accounts. As far as registry edits, I have already done them as per the documentation and allowed full control to my administrators group (which my glassfish account belongs to).

When using a local account with Windows Server 2008 R2 (64 bit my case) there are missing steps in the documentation. Windows does not allow remote access to c$ by default any longer which is required for the validate-dcom to work.

These steps are validated by using the windows command "net use" in the command prompt. The first example shows default windows behaviour attempting to access the c$ on a remote host. The second example illustrates what happens after an configuration change in Windows that isn't documented by Oracle:

<snip>
C:\Users\GFAdmin>net use \\remotehost\c$ /USER:remotehost\glassfish
The password is invalid for \\remotehost\c$.

Enter the password for 'remotehost\glassfish' to connect to 'remotehost':
System error 5 has occurred.

Access is denied.

C:\Users\GFAdmin>net use \\remotehost\c$ /USER:remotehost\glassfish
Enter the password for 'remotehost\glassfish' to connect to 'remotehost':
The command completed successfully.

C:\Users\GFAdmin>
</snip>

Once you perform the following steps to allow remote access to c$, DCOM works with a local account (almost):

Enable Local Account Token Filter Policy
• Click Start
• Type regedit in the Start Search box, and then click regedit in the Programs list.
• Expand the following subkey:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System
• If the LocalAccountTokenFilterPolicy registry entry does not exist, follow these steps:
a. On the Edit menu, click New, and then click DWORD Value
b. Type LocalAccountTokenFilterPolicy, and then press ENTER
• Right-click LocalAccountTokenFilterPolicy, and then click Modify
• In the Value data box, type 1, and then click OK
• Done, no need to reboot.

However now there is a new problem (PROGRESS!)

Successfully connected to Windows Shares at port 445 on host remotehost.
Successfully accessed C: on remotehost using DCOM.
Successfully wrote delete_me.bat to C: on remotehost using DCOM.
Could not connect to WMI (Windows Management Interface) on remotehost. : Error setting up remote connection to WMI

A quick search shows that Byron has run into this before:
http://java.net/projects/glassfish/lists/commits/archive/2011-11/message/143

At this point I have done a lot of troubleshooting to figure out the remote WMI failures which I cannot document here due to size. Needless to say, something is still either not setup properly or broken in the validation code that Byron is looking at.


Byron Nevins added a comment - 07/Mar/12 09:04 PM

Checked in a fix to the (first) problem where the windows domain was never used for auth/auth.
I checked it in to 4.0 just now and put the fix for 3.1.2 into the pipeline for the first patch release.

Sending common\src\main\java\com\sun\enterprise\util\cluster\windows\io\WindowsRemoteFileSystem.java
Transmitting file data .
Committed revision 52813.


easarina added a comment - 07/Mar/12 09:58 PM

When DCOM enabling problem will be solved, you can give in install-node-dcom a full machine name, i.e. <machne_name.domain>


easarina added a comment - 09/Mar/12 10:26 PM

Byron, could you give me the fix as a patch, so I can verify the fix, using a negative testcase.


Byron Nevins added a comment - 02/Apr/12 10:47 PM

BugDB 13917234


mtobler added a comment - 22/Jun/12 09:43 PM

Is there a release Date for this fix? I am trying to set up Clustering using DCOM on multiple 2008 R2 Servers and I consistently get the following:
asadmin> validate-dcom --passwordfile do-not-delete gf01
remote failure:
Successfully verified that the host, gf01, is not the local machine as required.
Successfully resolved host name to: gf01/172.18.11.169
Successfully connected to DCOM Port at port 135 on host gf01.
Successfully connected to NetBIOS Session Service at port 139 on host gf01.
Successfully connected to Windows Shares at port 445 on host gf01.
The remote file, C: doesn't exist on gf01 : Logon failure: unknown user name or bad password.

I have gone through every document I can find on this issue and have verified all settings/registry kays/etc are correct and this won't go away. I get the same via the console.


Byron Nevins added a comment - 26/Jun/12 08:36 PM

It *is* fixed in the open-source 4.0 codebase. You could always go grab the one changed file and build it. I commented on the filename and the subversion revision number in this issue. There is no date yet for the next 3.x version of GlassFish.


rdblaha1 added a comment - 04/Oct/13 08:48 PM

I do not see a reply to the comment mtobler 22/Jun/12. I receive a similar response for validate-dcom and can find no solution in an forum, blog, or tutorial. In this issue and GLASSFISH-18451 the trail ends June 2012. Where can I find an answer. (Glassfish 3.1.2.2, Java 1.6.0_45).

The only difference for what I am getting returned is the last lines.

Using command line:

C:\glassfish\glassfish3122\glassfish\bin>.\asadmin --port 4892 --passwordfile C:\glassfish\glassfish3122\glassfish\domains\testing_cluster\config\dcom-passwords validate-dcom -w Rick.Blaha -v=true TestWeb2
remote failure:
Successfully verified that the host, TestWeb2, is not the local machine as required.
Successfully resolved host name to: TestWeb2/10.3.30.129
Successfully connected to DCOM Port at port 135 on host TestWeb2.
Successfully connected to NetBIOS Session Service at port 139 on host TestWeb2.
Successfully connected to Windows Shares at port 445 on host TestWeb2.
dcom.no.remote.file.access : Logon failure: unknown user name or bad password.

Command validate-dcom failed.

I only need direction to the document solution whether found in another issue or wherever. If I am completely out of the ballpark tell me that too.