Dec 02

Service Response Time

There are several ways to get this done but I found using curl gives a good quick test of how long a web response was. You can use a formatted text file or just command line.

Command line can be something like this:

# curl -w "\ntime_namelookup:\t%{time_namelookup}\ntime_connect:\t\t%{time_connect}\ntime_appconnect:\t%{time_appconnect}\ntime_pretransfer:\t%{time_pretransfer}\ntime_redirect:\t\t%{time_redirect}\ntime_starttransfer:\t%{time_starttransfer}\n----------\ntime_total:\t\t%{time_total}\n" -o /dev/null -s https://www.wormly.com
 
time_namelookup:        0.366
time_connect:           0.367
time_appconnect:        0.402
time_pretransfer:       0.402
time_redirect:          0.000
time_starttransfer:     0.408
----------
time_total:             0.408

Or using a text file for formatting:

# curl -w "@curl-format.txt" -o /dev/null -s https://www.wormly.com

            time_namelookup:  0.127
               time_connect:  0.128
            time_appconnect:  0.142
           time_pretransfer:  0.142
              time_redirect:  0.000
         time_starttransfer:  0.150
                            ----------
                 time_total:  0.151

Text file like this:

# cat curl-format.txt 
\n
            time_namelookup:  %{time_namelookup}\n
               time_connect:  %{time_connect}\n
            time_appconnect:  %{time_appconnect}\n
           time_pretransfer:  %{time_pretransfer}\n
              time_redirect:  %{time_redirect}\n
         time_starttransfer:  %{time_starttransfer}\n
                            ----------\n
                 time_total:  %{time_total}\n
\n

Comments Off on Service Response Time
comments

Dec 01

Linux tabbed SSH connection manager

I like to work in a tabbed SSH connection manager. Especially when I have hundred's or thousands of machines to connect to. A connection manager like putty keeps track of machine names and login info. Using a tabbed interface like MTPutty can make your life a whole lot easier with treeview/groups and side by side terminals. And additionally if you can cluster the terminal commands it can be an added bonus.

So far I have not really liked anything in the Linux world as far as a SSH connection manager. Ubuntu does come with putty which seems to work the same as in the Windows world. Best I could find is an application written in Python called Gnome Connection Manager (gcm). In the Ubuntu 15.10 repos the package is called gnome-connection-manager. Be warned of a few things:

1. Seems the code is at least a few years old and the website does not seem to have documentation or any kind of discussion. The code is all python so you can look at fixing and emailing the owner.
2. Be sure to check the paste-right-click in ~/.gcm/gcm.conf. This can be a nasty setting if you did not expect it. I copy and paste a lot between a Windows desktop and a Linux guest and almost accidentally pasted garbage into a critical device.
3. Also check auto-copy-selection if you like putty style behavior where anything selected in your SSH terminal should be in the copy buffer.

I think if the developer put a little bit more love into gnome-connection-manager it would definitely be a keeper and first rate gnome app. I have looked at some other options like hotssh but worth checking out is Remmina. Unfortunately for me Remmina was very buggy.

Comments Off on Linux tabbed SSH connection manager
comments

Nov 11

Icinga2 on Solaris 11

I typically prefer using Nagios for network monitoring. Nagios itself is tricky to get going on Solaris and I have had a long running issue with Nagios on Solaris. Despite the issue around Nagios Core Worker timing out I still use Nagios on Solaris. If Linux is an option in a particular environment it would be preferable to use a packaged up Nagios from any of the popular distributions.

Having said that I have recently tried Icinga2 on Solaris and here is some notes around getting it running. Consider yourself warned this is not pretty but it runs surprisingly well and lightweight compared to Nagios. At least on Solaris SPARC.

There is a few hoops to jump through before compiling icinga2 like the compile environment and boost is particularly nasty. I will document that separate at some point in future. As I said you are on your own to get the compile environment and pre-requisites in order.

Stage and compile:

# pwd
/usr/src
# wget https://github.com/Icinga/icinga2/archive/v2.3.11.tar.gz
# gzip -d v2.3.11.tar.gz 
# tar xf v2.3.11.tar 
# mv v2.3.11.tar icinga-v2.3.11.tar
# cd icinga2-2.3.11/
# mkdir build && cd build
# cmake -D MYSQL_INCLUDE_DIR=/usr/mysql/5.1/include -D MYSQL_LIB=/usr/mysql/5.1/lib -DICINGA2_WITH_PGSQL=OFF ..

Issue 1 -- cmake:

# pwd
/usr/src/icinga2-2.3.11
# diff CMakeLists.txt CMakeLists.txt.orig 
135,136c135,136
<     set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g -pthread -lm")
<     set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g -pthread -DSOLARIS2=11 -D_POSIX_PTHREAD_SEMANTICS")
---
>     set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g")
>     set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g")
139,140c139,140
<   set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl")
<   set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl")
---
>   set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--gc-sections")
>   set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,--gc-sections")

** rerun cmake after fixing CMakeLists.txt

Compile issue 2 -- INFINITY:

# cd build/
# gmake VERBOSE=1

[..]
/usr/src/icinga2-2.3.11/lib/base/math-script.cpp:85:18: error: ‘INFINITY’ was not declared in this scope

# diff /usr/src/icinga2-2.3.11/lib/base/math-script.cpp /usr/src/icinga2-2.3.11/lib/base/math-script.cpp.orig
30,31d29
< #define INFINITY (__builtin_huge_val ())
< 

# gmake VERBOSE=1

** takes a very long time

Compile issue 3 -- mysql.h:

/usr/src/icinga2-2.3.11/lib/db_ido_mysql/idomysqlconnection.hpp:27:19: fatal error: mysql.h: No such file or directory

# find / -name mysql.h
/usr/mysql/5.1/include/mysql/mysql.h
/usr/mysql/5.5/include/mysql.h

# cmake -D MYSQL_INCLUDE_DIR=/usr/mysql/5.5/include -D MYSQL_LIB=/usr/mysql/5.1/lib -DICINGA2_WITH_PGSQL=OFF ..
-- Could NOT find yajl (missing:  YAJL_LIBRARY YAJL_INCLUDE_DIR) 
running /usr/bin/cmake -E copy_if_different "/usr/src/icinga2-2.3.11/third-party/yajl/src/api/yajl_parse.h" "/usr/src/icinga2-2.3.11/build/third-party/yajl/src/../include/yajl"  2>&1
running /usr/bin/cmake -E copy_if_different "/usr/src/icinga2-2.3.11/third-party/yajl/src/api/yajl_gen.h" "/usr/src/icinga2-2.3.11/build/third-party/yajl/src/../include/yajl"  2>&1
running /usr/bin/cmake -E copy_if_different "/usr/src/icinga2-2.3.11/third-party/yajl/src/api/yajl_common.h" "/usr/src/icinga2-2.3.11/build/third-party/yajl/src/../include/yajl"  2>&1
running /usr/bin/cmake -E copy_if_different "/usr/src/icinga2-2.3.11/third-party/yajl/src/api/yajl_tree.h" "/usr/src/icinga2-2.3.11/build/third-party/yajl/src/../include/yajl"  2>&1
-- MySQL Include dir: /usr/mysql/5.5/include  library dir: /usr/mysql/5.1
-- MySQL client libraries: mysqlclient_r
-- Configuring done
-- Generating done
-- Build files have been written to: /usr/src/icinga2-2.3.11/build

** of course rerun gmake

Install:

# gmake install
[  1%] Built target mmatch
[ 10%] Built target yajl
[..]
-- Installing: /usr/local/share/doc/icinga2/markdown/21-debug.md
-- Installing: /usr/local/share/doc/icinga2/markdown/12-distributed-monitoring-ha.md

Attempt first run:

# LD_LIBRARY_PATH=/usr/local/lib/:/usr/mysql/5.1/lib/mysql
# echo $LD_LIBRARY_PATH
/usr/local/lib/:/usr/mysql/5.1/lib/mysql
# /usr/local/sbin/icinga2 daemon
[2015-11-10 06:59:17 -0800] information/cli: Icinga application loader (version: r2.3.11-1)
[2015-11-10 06:59:17 -0800] information/cli: Loading application type: icinga/IcingaApplication
[..]
[2015-11-10 06:59:17 -0800] information/ConfigItem: Checked 1 UserGroup(s).
[2015-11-10 06:59:17 -0800] information/ConfigItem: Checked 1 IcingaApplication(s).
[2015-11-10 06:59:17 -0800] information/ConfigItem: Checked 1 ScheduledDowntime(s).
[2015-11-10 06:59:17 -0800] information/ScriptGlobal: Dumping variables to file '/usr/local/var/cache/icinga2/icinga2.vars'
[2015-11-10 06:59:17 -0800] information/DynamicObject: Restoring program state from file '/usr/local/var/lib/icinga2/icinga2.state'
[2015-11-10 06:59:17 -0800] information/DynamicObject: Restored 157 objects. Loaded 5 new objects without state.
[2015-11-10 06:59:17 -0800] information/ConfigItem: Triggering Start signal for config items
[2015-11-10 06:59:17 -0800] information/DbConnection: Resuming IDO connection: ido-mysql
[2015-11-10 06:59:17 -0800] information/ConfigItem: Activated all objects.
[2015-11-10 06:59:17 -0800] information/IdoMysqlConnection: MySQL IDO instance id: 1 (schema version: '1.13.0')

Web front-end:

Definitely read:
https://github.com/Icinga/icingaweb2
https://github.com/Icinga/icingaweb2/blob/master/doc/installation.md

# pwd
/var/apache2/2.2/htdocs
# mv master.zip icinga2_master_old.zip
# wget https://github.com/Icinga/icingaweb2/archive/master.zip

# unzip icinga2_master_20151110.zip
[..]
 inflating: icingaweb2-master/test/php/res/status/icinga.objects.cache  
  inflating: icingaweb2-master/test/php/res/status/icinga.status.dat  

Since I am upgrading in my case I followed header "Upgrading Icinga Web 2"

Before upgrade version.

# pwd
/var/apache2/2.2/htdocs/icingaweb2
# more VERSION 
v2.0.0-rc1

# mv icingaweb2 icingaweb2.20151110
# mv icingaweb2-master/ icingaweb2

Not sure if my upgrade actually needed the schema change(did not verify) this but I ran it just in case.

# pwd
/var/apache2/2.2/htdocs/icingaweb2.new/etc/schema/mysql-upgrades#
# mysql -u root -p icinga < 2.0.0beta3-2.0.0rc1.sql 

** In addition although the v2.0.0-rc1 web front-end is working fine for me the latest (v2.0.0) is not. I have a problem on the dashboard I have not had time to track down. The error is: The filter column "view" is not allowed here.

Comments Off on Icinga2 on Solaris 11
comments

Nov 07

SSH Key Authentication insist on password

I recently spent a lot of time trying to figure out why a private key I exported from putty was not working right on Linux. My issue was I kept getting prompted for a password no matter what.

One clue that may or may not show the issue in the SSHD logs I was NOT seeing "debug1: ssh_rsa_verify: signature correct" when keyed authentication failed and switched to password.

I fixed my problem by deleting the public key in the Linux client .ssh folder. So most likely while copying keys from Windows and putty I must have gotten a mismatch on private/public pair.

If you need it you can re-create it as follow:
rosso@rrosso-VirtualBox:~/.ssh$ ssh-keygen -y -f id_rsa > id_rsa.pub

$ ssh root@usli-utility.mgmt.asg.ad
Last login: Sat Nov 7 14:14:08 2015 from 192.168.38.104
Oracle Corporation SunOS 5.11 11.3 September 2015
You have new mail.

Comments Off on SSH Key Authentication insist on password
comments

Sep 09

Powerline for your terminal on Ubuntu

I noticed on Fedora it is really easy to enable powerline in your terminal. This article explains it well: http://fedoramagazine.org/add-power-terminal-powerline/

On Ubuntu 15.04 it looks like powerline is pretty easy to enable although it did not appear like the tmux or vim powerline plugins are available from regular repos.

Enable powerline in your terminal like this:

# apt-get install powerline fonts-powerline

~ cat .bashrc
[..]
if [ -f `which powerline-daemon` ]; then
  powerline-daemon -q
  POWERLINE_BASH_CONTINUATION=1
  POWERLINE_BASH_SELECT=1
  . /usr/share/powerline/bindings/bash/powerline.sh
fi

3
comments

Aug 27

ZFS Storage Appliance RESTful API

Until now I have used ssh and javascript to do some of the more advanced automation tasks like snapshots, cloning and replication. I am starting to look at porting to REST and here is a quick example of two functions.

** I needed fabric and python-requests linux packages installed for python.

#!/usr/bin/env fab
 
from fabric.api import task,hosts,settings,env
from fabric.utils import abort
import requests, json, os
from datetime import date

#requests.packages.urllib3.disable_warnings()  
today = date.today()
 
# ZFSSA API URL
url = "https://192.168.2.200:215"
 
# ZFSSA authentication credentials, it reads username and password from environment variables ZFSUSER and ZFSPASSWORD
zfsauth = (os.getenv('ZFSUSER'), os.getenv('ZFSPASSWORD'))
 
jsonheader={'Content-Type': 'application/json'}

# This gets the pool list
def list_pools():
  r = requests.get("%s/api/storage/v1/pools" % (url), auth=zfsauth, verify=False, headers=jsonheader)
  if r.status_code != 200:
    abort("Error getting pools %s %s" % (r.status_code, r.text))
  j = json.loads(r.text) 
  #print j

  for pool in j["pools"]:
    #print pool
    #{u'status': u'online', u'profile': u'stripe', u'name': u'tank1', u'owner': u'zfsapp1', u'usage': {}, u'href': u'/api/storage/v1/pools/tank1', u'peer': u'00000000-0000-0000-0000-000000000000', u'asn': u'91bdcaef-fea5-e796-8793-f2eefa46200a'}ation
    print "pool: %s and status: %s" % (pool["name"], pool["status"])


# Create project
def create_project(pool, projname):
  # First check if the target project name already exists
  r = requests.get("%s/api/storage/v1/pools/%s/projects/%s" % (url, pool, projname), auth=zfsauth, verify=False, headers=jsonheader)
  if r.status_code != 404:
    abort("ZFS project %s already exists (or other error): %s" % (projname, r.status_code))

  payload = { 'name': projname, 'sharenfs': 'ro' }
  r = requests.post("%s/api/storage/v1/pools/%s/projects" % (url, pool), auth=zfsauth, verify=False, data=json.dumps(payload), headers=jsonheader)
  if r.status_code == 201:
    print "project created"
  else:
    abort("Error creating project %s %s" % (r.status_code, r.text))

print "\n\nTest list pools and create a project\n"
list_pools()
create_project('tank1','proj-01')

References:
http://www.oracle.com/technetwork/articles/servers-storage-admin/zfs-appliance-scripting-1508184.html
http://docs.oracle.com/cd/E51475_01/html/E52433/
http://ilmarkerm.blogspot.com/2014/12/sample-code-using-oracle-zfs-storage.html

1
comments

Aug 21

Solaris ipadm show-prop and returning only current value

A lot has been written about Solaris 11 networking changes and how to manage it with different commands. This is just a quick note on something specific to what I was trying. Previous versions of Solaris I could do a quick ndd command to get or set a value. For example:

# ndd -get /dev/tcp tcp_smallest_anon_port
9000

And yes above ndd command still works in Solaris 11.

In Solaris 11 most documentation would indicate doing the following:

# ipadm show-prop -p smallest_anon_port tcp
PROTO PROPERTY              PERM CURRENT      PERSISTENT   DEFAULT      POSSIBLE
tcp   smallest_anon_port    rw   9000         9000         32768        1024-65500

Above is fine if you are checking something but what if you need to just return the current value. You can of course do some string manipulation but if you dig a little deeper ipadm can return exactly what you want as follow. In my case I wanted to have a reliable check so my puppet manifest would not set the value again if already correct(using exec/onlyif in puppet).

Returning only the value with ipadm works as follow:

# ipadm show-prop -o CURRENT -c -p smallest_anon_port tcp
9000

I plan to add more about puppet and Solaris in general later but here is the ipadm check in the puppet manifest for reference.

exec { "ipadm smallest_anon_port tcp":
  command     => "ipadm set-prop -p smallest_anon_port=9000 tcp",
  path        => $execPath,
  onlyif      => "ipadm show-prop -o CURRENT -c -p smallest_anon_port tcp | grep -v 9000"
}

Comments Off on Solaris ipadm show-prop and returning only current value
comments

Jun 13

Firewalld Rich Rule

To add a so called rich rule to firewalld I did the following.

Check existing rules after a recent upgrade to Fedora 22. 55555/tcp was a custom app not Fedora default.

# firewall-cmd --get-default-zone
FedoraServer

# firewall-cmd --zone=FedoraServer --list-all
FedoraServer (default)
  interfaces:
  sources:
  services: cockpit dhcpv6-client http smtp ssh
  ports: 55555/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

Lets remove some rules.

# firewall-cmd --permanent --zone=FedoraServer --remove-port=55555/tcp
success
# firewall-cmd --permanent --zone=FedoraServer --remove-service=cockpit
success
# firewall-cmd --permanent --zone=FedoraServer --remove-service=dhcpv6-client
success

Add custom rule non-permanent.
** Note x.x.x.x is a placeholder for a real public IP. Most likely you won't need a public IP but a non-routable Class C or B on your internal network.

# firewall-cmd --zone=FedoraServer --add-rich-rule="rule family="ipv4" source address="x.x.x.x/32" port protocol="tcp" port="55555" accept"
success

Or if you need it permanently added remember to use --permanent.

Restart the firewall.

# systemctl restart firewalld.service
# firewall-cmd --zone=FedoraServer --list-all
FedoraServer (default)
  interfaces:
  sources:
  services: http smtp ssh
  ports:
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:
        rule family="ipv4" source address="x.x.x.x/32" port port="55555" protocol="tcp" accept

Comments Off on Firewalld Rich Rule
comments

May 14

Systemd using systemctl and legacy init scripts

I recently played with the Solus Linux distribution to check out budgie on the desktop. Solus(previously evolveOS) use systemd like most distributions now do. Of course anything I test is inside a VM and I wanted to get the Virtualbox Guest Additions to work. During the guest additions install from ISO it does complain about not knowing the distribution and for the most part when the OS is not recognized Virtualbox Guest Additions install then just stick the init scripts in /etc/init.d and let you figure it out.

Since I wanted to play with systemd a little bit here is what I did to make the guest additions scripts run at boot.  Here is a link with some comparisons between systemd and sysvinit: https://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet

Lets add some units so systemctl can do its work.

$ pwd
/usr/lib/systemd/system

$ cat vboxadd.service
[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/vboxadd
Description=LSB: VirtualBox Linux Additions kernel modules
Before=runlevel2.target runlevel3.target runlevel4.target runlevel5.target shutdown.target
Conflicts=shutdown.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/init.d/vboxadd start
ExecStop=/etc/init.d/vboxadd stop

[Install]
WantedBy=multi-user.target

Ok let's enable the unit.

$ systemctl enable vboxadd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/vboxadd.service to /usr/lib64/systemd/system/vboxadd.service.

$ systemctl status vboxadd.service
● vboxadd.service - LSB: VirtualBox Linux Additions kernel modules
   Loaded: loaded (/etc/init.d/vboxadd; enabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:systemd-sysv-generator(8)

Now start this service.

$ systemctl start vboxadd.service

And check it's status.

$ systemctl status vboxadd.service
● vboxadd.service - LSB: VirtualBox Linux Additions kernel modules
   Loaded: loaded (/etc/init.d/vboxadd; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2015-05-14 13:39:40 CDT; 2s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 610 ExecStart=/etc/init.d/vboxadd start (code=exited, status=0/SUCCESS)

May 14 13:39:40 solus vboxadd[610]: Starting the VirtualBox Guest Additions ...done.

You can now do the same for the other two init scripts namely vboxadd-service and vboxadd-x11.

$ cat vboxadd-service.service
[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/vboxadd-service
Description=LSB: VirtualBox Additions service
Before=runlevel2.target runlevel3.target runlevel4.target runlevel5.target shutdown.target
After=vboxadd.service
Conflicts=shutdown.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/init.d/vboxadd-service start
ExecStop=/etc/init.d/vboxadd-service stop

[Install]
WantedBy=multi-user.target

$ systemctl enable vboxadd-service.service
Created symlink from /etc/systemd/system/multi-user.target.wants/vboxadd-service.service to /usr/lib64/systemd/system/vboxadd-service.service.

$ systemctl start vboxadd-service.service

$ systemctl status vboxadd-service.service
● vboxadd-service.service - LSB: VirtualBox Additions service
   Loaded: loaded (/etc/init.d/vboxadd-service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2015-05-14 13:46:01 CDT; 1s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 614 ExecStart=/etc/init.d/vboxadd-service start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/vboxadd-service.service
           └─622 /usr/sbin/VBoxService

May 14 13:46:01 solus vboxadd-service[614]: Starting VirtualBox Guest Addition service  ...done.

$ cat vboxadd-x11.service
[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/vboxadd-x11
Description=LSB: VirtualBox Linux Additions kernel modules

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/init.d/vboxadd-x11 start
ExecStop=/etc/init.d/vboxadd-x11 stop

[Install]
WantedBy=multi-user.target

$ systemctl enable vboxadd-x11.service
Created symlink from /etc/systemd/system/multi-user.target.wants/vboxadd-x11.service to /usr/lib64/systemd/system/vboxadd-x11.service.

$ systemctl start vboxadd-x11.service

$ systemctl status vboxadd-x11.service
● vboxadd-x11.service - LSB: VirtualBox Linux Additions kernel modules
   Loaded: loaded (/etc/init.d/vboxadd-x11; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2015-05-14 13:47:04 CDT; 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 649 ExecStart=/etc/init.d/vboxadd-x11 start (code=exited, status=0/SUCCESS)

At this point after reboot looks like systemd ran the scripts.  Let's see if the vbox scripts/units show as a dependency for multi-user.

$ systemctl show -p "Wants" multi-user.target
Wants=systemd-logind.service systemd-update-utmp-runlevel.service NetworkManager.service dbus.service getty.target vboxadd.service vboxadd-x11.service systemd-resolved.service vboxadd-service.service remote-fs.target systemd-ask-password-wall.path systemd-user-sessions.service

Comments Off on Systemd using systemctl and legacy init scripts
comments

Apr 27

Solaris Information On WWN

Recently I had some messages on a T5-2 hypervisor and I needed to find out exactly what it is complaining about.

Messages looked like this:

fctl: [ID 517869 kern.warning] WARNING: fp(5)::GPN_ID for D_ID=160003 failed
fctl: [ID 517869 kern.warning] WARNING: fp(5)::N_x Port with D_ID=160003, PWWN=23220002ac0012b4 disappeared from fabric
fctl: [ID 517869 kern.warning] WARNING: fp(6)::GPN_ID for D_ID=c0001 failed
fctl: [ID 517869 kern.warning] WARNING: fp(6)::N_x Port with D_ID=c0001, PWWN=23210002ac0012b4 disappeared from fabric
fctl: [ID 517869 kern.warning] WARNING: fp(6)::N_x Port with D_ID=c0001, PWWN=23210002ac0012b4 reappeared in fabric
fctl: [ID 517869 kern.warning] WARNING: fp(5)::N_x Port with D_ID=160003, PWWN=23220002ac0012b4 reappeared in fabric

First let's check the adapters and ports.

# fcinfo hba-port
HBA Port WWN: 21000024ff4e2a9c
        Port Mode: Initiator
        Port ID: 210600
        OS Device Name: /dev/cfg/c4
        Manufacturer: QLogic Corp.
        Model: 371-4325-02
        Firmware Version: 5.08.00
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.03; EFI: 2.01;
        Serial Number: 0402T00-1315130088
        Driver Name: qlc
        Driver Version: 20131114-4.03
        Type: N-port
        State: online
        Supported Speeds: 2Gb 4Gb 8Gb
        Current Speed: 8Gb
        Node WWN: 20000024ff4e2a9c
        Max NPIV Ports: 255
        NPIV port list:
HBA Port WWN: 21000024ff4e2a9d
        Port Mode: Initiator
        Port ID: 0
        OS Device Name: /dev/cfg/c5
        Manufacturer: QLogic Corp.
        Model: 371-4325-02
        Firmware Version: 5.08.00
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.03; EFI: 2.01;
        Serial Number: 0402T00-1315130088
        Driver Name: qlc
        Driver Version: 20131114-4.03
        Type: unknown
        State: offline
        Supported Speeds: 2Gb 4Gb 8Gb
        Current Speed: not established
        Node WWN: 20000024ff4e2a9d
        Max NPIV Ports: 255
        NPIV port list:
HBA Port WWN: 21000024ff4e29d6
        Port Mode: Initiator
        Port ID: 660700
        OS Device Name: /dev/cfg/c6
        Manufacturer: QLogic Corp.
        Model: 371-4325-02
        Firmware Version: 5.08.00
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.03; EFI: 2.01;
        Serial Number: 0402T00-1315129988
        Driver Name: qlc
        Driver Version: 20131114-4.03
        Type: N-port
        State: online
        Supported Speeds: 2Gb 4Gb 8Gb
        Current Speed: 8Gb
        Node WWN: 20000024ff4e29d6
        Max NPIV Ports: 255
        NPIV port list:
HBA Port WWN: 21000024ff4e29d7
        Port Mode: Initiator
        Port ID: 0
        OS Device Name: /dev/cfg/c7
        Manufacturer: QLogic Corp.
        Model: 371-4325-02
        Firmware Version: 5.08.00
        FCode/BIOS Version:  BIOS: 2.02; fcode: 2.03; EFI: 2.01;
        Serial Number: 0402T00-1315129988
        Driver Name: qlc
        Driver Version: 20131114-4.03
        Type: unknown
        State: offline
        Supported Speeds: 2Gb 4Gb 8Gb
        Current Speed: not established
        Node WWN: 20000024ff4e29d7
        Max NPIV Ports: 255
        NPIV port list:

Let's check which ports are connected.

# luxadm -e port
/devices/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0:devctl        CONNECTED
/devices/pci@340/pci@1/pci@0/pci@5/SUNW,qlc@0/fp@0,0:devctl        CONNECTED
/devices/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0,1/fp@0,0:devctl      NOT CONNECTED
/devices/pci@340/pci@1/pci@0/pci@5/SUNW,qlc@0,1/fp@0,0:devctl      NOT CONNECTED

Now let's find devices on the above two connected ports. You should be able to see the original warning WWN's listed here.

# luxadm -e dump_map /devices/pci@340/pci@1/pci@0/pci@4/SUNW,qlc@0/fp@0,0:devctl
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    160000  0         20220002ac0012b4 2ff70002ac0012b4 0x0  (Disk device)
1    160001  0         21220002ac0012b4 2ff70002ac0012b4 0x0  (Disk device)
2    160002  0         22220002ac0012b4 2ff70002ac0012b4 0x0  (Disk device)
3    160003  0         23220002ac0012b4 2ff70002ac0012b4 0x0  (Disk device)	<-- Messages show this one
4    210900  0         21000024ff57d60d 20000024ff57d60d 0x0  (Disk device)
5    210a00  0         21000024ff57d64d 20000024ff57d64d 0x0  (Disk device)
6    210b00  0         21000024ff57d649 20000024ff57d649 0x0  (Disk device)
7    210c00  0         21000024ff57d60b 20000024ff57d60b 0x0  (Disk device)
8    210600  0         21000024ff4e2a9c 20000024ff4e2a9c 0x1f (Unknown Type,Host Bus Adapter)

# luxadm -e dump_map /devices/pci@340/pci@1/pci@0/pci@5/SUNW,qlc@0/fp@0,0:devctl
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    c0001   0         23210002ac0012b4 2ff70002ac0012b4 0x0  (Disk device)	<-- Messages show this one
1    c0002   0         22210002ac0012b4 2ff70002ac0012b4 0x0  (Disk device)
2    c0003   0         21210002ac0012b4 2ff70002ac0012b4 0x0  (Disk device)
3    c0004   0         20210002ac0012b4 2ff70002ac0012b4 0x0  (Disk device)
4    660000  0         21000024ff57d64c 20000024ff57d64c 0x0  (Disk device)
5    660100  0         21000024ff57d60a 20000024ff57d60a 0x0  (Disk device)
6    660200  0         21000024ff57d648 20000024ff57d648 0x0  (Disk device)
7    660a00  0         21000024ff57d60c 20000024ff57d60c 0x0  (Disk device)
8    660700  0         21000024ff4e29d6 20000024ff4e29d6 0x1f (Unknown Type,Host Bus Adapter)

You can try and see if the device is listed here. In my case it was not so it appears that we are not using this device but maybe the switch still have us zoned for the device.

# luxadm probe
No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):
  Node WWN:20000024ff57d64d  Device Type:Disk device
    Logical Path:/dev/rdsk/c0t600144F086479F15000053DA5A03000Ad0s2
[..]

# luxadm probe | grep 2ac0012b4  <-- Not found so remote?

We probably did not need all steps above to get to the next command but I am listing everything since it depends on the situation and what you are tracing.

Here I can see the relevant WWN is a 3PAR SAN and in my case I knew I am not using 3PAR any longer so I can check if this is a zoning issue.

# fcinfo remote-port -ls -p 21000024ff4e29d6
Remote Port WWN: 20210002ac0012b4
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port Symbolic Name: 1404788 - 0:2:1 - LPe12004
        Node WWN: 2ff70002ac0012b4
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 12
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 48
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES
          OS Device Name: /dev/es/ses0
Remote Port WWN: 21210002ac0012b4
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port Symbolic Name: 1404788 - 1:2:1 - LPe12004
        Node WWN: 2ff70002ac0012b4
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 6
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 32
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES
          OS Device Name: /dev/es/ses1
Remote Port WWN: 22210002ac0012b4
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port Symbolic Name: 1404788 - 2:2:1 - LPe12004
        Node WWN: 2ff70002ac0012b4
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 16
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES
          OS Device Name: /dev/es/ses2
Remote Port WWN: 23210002ac0012b4
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port Symbolic Name: 1404788 - 3:2:1 - LPe12004
        Node WWN: 2ff70002ac0012b4
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 16
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES
          OS Device Name: /dev/es/ses3
[..]

# fcinfo remote-port -ls -p 21000024ff4e2a9c
Remote Port WWN: 20220002ac0012b4
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port Symbolic Name: 1404788 - 0:2:2 - LPe12004
        Node WWN: 2ff70002ac0012b4
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 6
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 32
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES
          OS Device Name: /dev/es/ses4
Remote Port WWN: 21220002ac0012b4
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port Symbolic Name: 1404788 - 1:2:2 - LPe12004
        Node WWN: 2ff70002ac0012b4
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 16
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES
          OS Device Name: /dev/es/ses5
Remote Port WWN: 22220002ac0012b4
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port Symbolic Name: 1404788 - 2:2:2 - LPe12004
        Node WWN: 2ff70002ac0012b4
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 16
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES
          OS Device Name: /dev/es/ses6
Remote Port WWN: 23220002ac0012b4
        Active FC4 Types: SCSI
        SCSI Target: yes
        Port Symbolic Name: 1404788 - 3:2:2 - LPe12004
        Node WWN: 2ff70002ac0012b4
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 16
                Invalid CRC Count: 0
        LUN: 254
          Vendor: 3PARdata
          Product: SES
          OS Device Name: /dev/es/ses7

1
comments