updatecenter2
  1. updatecenter2
  2. UPDATECENTER2-2218

S11: intermitten connection refused while executing pkg command over a pkg.depotd running locally

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.3
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Solaris sparc 5.11, UC version = 1.122.2-50.2809

      Description

      To summarize the problem, "pkg install" fails sporadically with the following error message: Unable to contact a valid package depot.

      I've tried to get more info about the network issue, I've located the python code that prints "Unable to contact a valid package depot" (hooked pkg/vendor-packages/pkg/client/transport/transport.py) and hooked it in order to display more info on the exception:
      > print "TransportException: ",sys.exc_value

      The exception's value is "Framework error: code: 7 reason: Failed connect to localhost:16500; Connection refused.

      Based on Joe's inputs, we've tried to increased the thread number of pkg.depotd up to 100 (-s 100) without success.

      Also based on Joe inputs, we search for TIME_WAIT connection using netstat:

      # netstat -a -P tcp | grep TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.48467 139264      0 130880      0 TIME_WAIT
      sc11d1903.us.oracle.com.49218 pd-bigip-adc-prod2-vip14.us.oracle.com.4889  4875      0 129220      0 TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.36596 139264      0 130880      0 TIME_WAIT
      localhost.38025      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38987      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53725      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.51251      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43529      localhost.10000      130880      0 139060      0 TIME_WAIT
      localhost.58888      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36070      localhost.10000      130880      0 139264      0 TIME_WAIT
      # netstat -a -P tcp | grep TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.48467 139264      0 130880      0 TIME_WAIT
      sc11d1903.us.oracle.com.49218 pd-bigip-adc-prod2-vip14.us.oracle.com.4889  4875      0 129220      0 TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.36596 139264      0 130880      0 TIME_WAIT
      localhost.38025      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38987      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53725      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.51251      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43529      localhost.10000      130880      0 139060      0 TIME_WAIT
      localhost.58888      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36070      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.44351      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43583      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56914      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38282      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36781      localhost.10000      130880      0 139264      0 TIME_WAIT
      # netstat -a -P tcp | grep TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.48467 139264      0 130880      0 TIME_WAIT
      sc11d1903.us.oracle.com.49218 pd-bigip-adc-prod2-vip14.us.oracle.com.4889  4875      0 129220      0 TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.36596 139264      0 130880      0 TIME_WAIT
      localhost.38025      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38987      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53725      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.51251      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43529      localhost.10000      130880      0 139060      0 TIME_WAIT
      localhost.58888      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36070      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.44351      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43583      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56914      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38282      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36781      localhost.10000      130880      0 139264      0 TIME_WAIT
      # netstat -a -P tcp | grep TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.48467 139264      0 130880      0 TIME_WAIT
      sc11d1903.us.oracle.com.49218 pd-bigip-adc-prod2-vip14.us.oracle.com.4889  4875      0 129220      0 TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.36596 139264      0 130880      0 TIME_WAIT
      localhost.38025      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38987      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53725      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.51251      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43529      localhost.10000      130880      0 139060      0 TIME_WAIT
      localhost.58888      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36070      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.44351      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43583      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56914      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38282      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36781      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53773      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56007      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.50010      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.33758      localhost.10000      130880      0 139264      0 TIME_WAIT
      # netstat -a -P tcp | grep TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.48467 139264      0 130880      0 TIME_WAIT
      sc11d1903.us.oracle.com.49218 pd-bigip-adc-prod2-vip14.us.oracle.com.4889  4875      0 129220      0 TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.36596 139264      0 130880      0 TIME_WAIT
      localhost.38025      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38987      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53725      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.51251      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43529      localhost.10000      130880      0 139060      0 TIME_WAIT
      localhost.58888      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36070      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.44351      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43583      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56914      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38282      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36781      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53773      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56007      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.50010      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.33758      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.58936      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56176      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.62743      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.60881      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.63530      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.61656      localhost.10000      130880      0 139264      0 TIME_WAIT
      # netstat -a -P tcp | grep TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.48467 139264      0 130880      0 TIME_WAIT
      sc11d1903.us.oracle.com.49218 pd-bigip-adc-prod2-vip14.us.oracle.com.4889  4875      0 129220      0 TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.36596 139264      0 130880      0 TIME_WAIT
      localhost.38025      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38987      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53725      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.51251      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43529      localhost.10000      130880      0 139060      0 TIME_WAIT
      localhost.58888      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36070      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.44351      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43583      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56914      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38282      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36781      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53773      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56007      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.50010      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.33758      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.58936      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56176      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.62743      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.60881      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.63530      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.61656      localhost.10000      130880      0 139264      0 TIME_WAIT
      # netstat -a -P tcp | grep TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.48467 139264      0 130880      0 TIME_WAIT
      sc11d1903.us.oracle.com.49218 pd-bigip-adc-prod2-vip14.us.oracle.com.4889  4875      0 129220      0 TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.36596 139264      0 130880      0 TIME_WAIT
      localhost.38025      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38987      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53725      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.51251      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43529      localhost.10000      130880      0 139060      0 TIME_WAIT
      localhost.58888      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36070      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.44351      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43583      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56914      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38282      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36781      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53773      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56007      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.50010      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.33758      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.58936      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56176      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.62743      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.60881      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.63530      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.61656      localhost.10000      130880      0 139264      0 TIME_WAIT
      # netstat -a -P tcp | grep TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.48467 139264      0 130880      0 TIME_WAIT
      sc11d1903.us.oracle.com.49218 pd-bigip-adc-prod2-vip14.us.oracle.com.4889  4875      0 129220      0 TIME_WAIT
      sc11d1903.us.oracle.com.42830 sc11d1903.us.oracle.com.36596 139264      0 130880      0 TIME_WAIT
      localhost.38025      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38987      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53725      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.51251      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43529      localhost.10000      130880      0 139060      0 TIME_WAIT
      localhost.58888      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36070      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.44351      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.43583      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56914      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.38282      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.36781      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.53773      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56007      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.50010      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.33758      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.58936      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.56176      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.62743      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.60881      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.63530      localhost.10000      130880      0 139264      0 TIME_WAIT
      localhost.61656      localhost.10000      130880      0 139264      0 TIME_WAIT
      

      I then tried to decrease the TIME_WAIT_TIMEOUT using:

      # /usr/sbin/ndd -set /dev/tcp tcp_time_wait_interval 1000
      

      Unfortunately it did not fix the issue.

      1. mvn-debug-output.txt
        74 kB
        Romain Grécourt
      2. test-distro.sh
        2 kB
        Romain Grécourt

        Activity

        Hide
        Romain Grécourt added a comment -

        I can't attach the zip of the uc repo, it's too big

        Show
        Romain Grécourt added a comment - I can't attach the zip of the uc repo, it's too big
        Hide
        Romain Grécourt added a comment -

        attaching the output of the maven build (mvn -X).

        Show
        Romain Grécourt added a comment - attaching the output of the maven build (mvn -X).

          People

          • Assignee:
            Unassigned
            Reporter:
            Romain Grécourt
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated: