Amazon Linux 2 Image and LAMP

I recently migrated a LAMP server from Amazon Linux to an Amazon Linux 2 image.  Several reasons for why I needed this including it has systemd.

More here: https://aws.amazon.com/amazon-linux-2/

High level steps around mysql database, wordpress and static html migration was pretty smooth as I have done this multiple times. The only notable things to report on were:
1. You are probably going from a php5.x world to php7.x world and that could cause a few problems. In my case some older php gallery software threw multiple DEPRECATED problem so I had to work through them case by case.
2. I had a problem with php and mpm.
3. Certbot/Let’s Encrypt does not recognize Amazon Linux 2 from /etc/issue and fails.

LAMP Install:

Pretty much followed this without issues.

# yum update -y
# amazon-linux-extras install lamp-mariadb10.2-php7.2
# yum install -y httpd php mariadb-server php-mysqlnd
# systemctl enable httpd
# usermod -a -G apache ec2-user
# chown -R ec2-user:apache /var/www
# chmod 2775 /var/www && find /var/www -type d -exec sudo chmod 2775 {} \;
# find /var/www -type f -exec sudo chmod 0664 {} \;
# echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php

MPM Issue:

There may be other or better ways to solve this I have not had time to investigate further.

# systemctl start httpd
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

# systemctl status httpd.service -l
● httpd.service - The Apache HTTP Server
   Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/httpd.service.d
           └─php-fpm.conf
   Active: failed (Result: exit-code) since Tue 2018-05-29 13:35:34 UTC; 1min 21s ago
     Docs: man:httpd.service(8)
  Process: 12701 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
 Main PID: 12701 (code=exited, status=1/FAILURE)

May 29 13:35:34 ip-172-31-48-7.ec2.internal systemd[1]: Starting The Apache HTTP Server...
May 29 13:35:34 ip-172-31-48-7.ec2.internal httpd[12701]: [Tue May 29 13:35:34.378884 2018] [php7:crit] [pid 12701:tid 140520257956032] Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe.  You need to recompile PHP.
May 29 13:35:34 ip-172-31-48-7.ec2.internal httpd[12701]: AH00013: Pre-configuration failed

# pwd
/etc/httpd/conf.modules.d

# cp 00-mpm.conf /tmp
# vi 00-mpm.conf 
# diff 00-mpm.conf /tmp/00-mpm.conf 
11c11
< LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
---
> #LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
23c23
< #LoadModule mpm_event_module modules/mod_mpm_event.so
---
> LoadModule mpm_event_module modules/mod_mpm_event.so

# systemctl restart httpd

# ps -ef | grep http
root      9735     1  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9736  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9737  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9738  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9739  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND
apache    9740  9735  0 13:42 ?        00:00:00 /usr/sbin/httpd -DFOREGROUND

CERTBOT:

On the old server delete certs.

# /opt/eff.org/certbot/venv/local/bin/certbot delete
[..]
-------------------------------------------------------------------------------
Deleted all files relating to certificate blog.domain.com.
-------------------------------------------------------------------------------

On the new server install certs.

# yum install mod_ssl

# wget https://dl.eff.org/certbot-auto
# chmod a+x certbot-auto 
# ./certbot-auto --debug

Sorry, I don't know how to bootstrap Certbot on your operating system!

Work around the fact that certbot does not know about Amazon Linux 2 yet.

# yum install python-virtualenv python-augeas
# ./certbot-auto --debug --no-bootstrap
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Error while running apachectl configtest.

AH00526: Syntax error on line 100 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/pki/tls/certs/localhost.crt' does not exist or is empty


How would you like to authenticate and install certificates?
-------------------------------------------------------------------------------
1: Apache Web Server plugin - Beta (apache) [Misconfigured]
2: Nginx Web Server plugin - Alpha (nginx)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1

-------------------------------------------------------------------------------
The selected plugin encountered an error while parsing your server configuration
and cannot be used. The error was:

Error while running apachectl configtest.

AH00526: Syntax error on line 100 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/pki/tls/certs/localhost.crt' does not exist or is
empty

Have to fix ssl first apparently certbot need a generic localhost cert.

# openssl req -new -x509 -nodes -out localhost.crt -keyout localhost.key

# mv localhost.crt localhost.key /etc/pki/tls/certs/
# mv /etc/pki/tls/certs/localhost.key /etc/pki/tls/private/

# systemctl restart httpd

Now try again.

# ./certbot-auto --debug --no-bootstrap
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate and install certificates?
-------------------------------------------------------------------------------
1: Apache Web Server plugin - Beta (apache)
2: Nginx Web Server plugin - Alpha (nginx)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator apache, Installer apache
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel): E@MAIL.com
[..]

Which names would you like to activate HTTPS for?
-------------------------------------------------------------------------------
1: blog.domain.com
-------------------------------------------------------------------------------
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for blog.domain.com
Waiting for verification...
Cleaning up challenges
Created an SSL vhost at /etc/httpd/conf.d/vhost-le-ssl.conf
Deploying Certificate to VirtualHost /etc/httpd/conf.d/vhost-le-ssl.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.
-------------------------------------------------------------------------------
1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting vhost in /etc/httpd/conf.d/vhost.conf to ssl vhost in /etc/httpd/conf.d/vhost-le-ssl.conf

-------------------------------------------------------------------------------
Congratulations! You have successfully enabled https://blog.domain.com

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=blog.domain.com
-------------------------------------------------------------------------------
[..]

Test your site here:
https://www.ssllabs.com/ssltest/analyze.html?d=blog.domain.com&latest

Linux Routing Two Interfaces on Same Subnet

It’s possible you will never need to do this and most likely there are experts that say avoid doing this. I recently had a challenge to do exactly this so I am recording my effort for future reference. This link helped me get it to work: https://access.redhat.com/solutions/30564

My setup is similar to the link above but a few more layers. My setup is a Centos7 VM under KVM. KVM using SR-IOV Network Virtual Functions. And to further complicate it the KVM hypervisor is an Oracle Cloud (OCI) bare metal server. OCI hands out additional public IP addresses using VNIC’s which are added to the host via pass through. Out of scope here is adding VNIC’s to KVM guests. Also note the public IP is natted to private IP’s.

[root@centos7 opc]# cat /etc/iproute2/rt_tables 
[..]
100 t1
101 t2

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/route-ens3
10.1.0.0/16 dev ens3 src 10.1.1.12 table t1
default via 10.1.1.1 dev ens3 table t1

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/route-ens9
10.1.0.0/16 dev ens9 src 10.1.1.13 table t2
default via 10.1.1.1 dev ens9 table t2

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/rule-ens3
table t1 from 10.1.1.12

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/rule-ens9
table t2 from 10.1.1.13

Note may not need all below settings for example ens3 and ens9. Defaults may be enough.

[root@centos7 opc]# cat /etc/sysctl.d/99-sysctl.conf 
[..]
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_announce = 2

net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.ens3.rp_filter = 2
net.ipv4.conf.ens9.rp_filter = 2

Had some issues with /etc/sysconfig/network-scripts/route-ens* script not working at reboots, but manually running /etc/sysconfig/network-scripts/route-ens3 and route-ens9 worked. Commented DEFROUTE and GATEWAY and added NM_CONTROLLED=no and then routes worked at boot up.

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/ifcfg-ens3
TYPE=Ethernet
BOOTPROTO=static
#DEFROUTE=yes
NAME=ens3
DEVICE=ens3
ONBOOT=yes
IPADDR=10.1.1.12
NETMASK=255.255.255.0
#GATEWAY=10.1.1.1
NM_CONTROLLED="no"

[root@centos7 opc]# cat /etc/sysconfig/network-scripts/ifcfg-ens9
TYPE=Ethernet
BOOTPROTO=static
#DEFROUTE=yes
NAME=ens9
DEVICE=ens9
ONBOOT=yes
IPADDR=10.1.1.13
NETMASK=255.255.255.0
#GATEWAY=10.1.1.1
NM_CONTROLLED="no"

Reboot

[opc@centos7 ~]$ sudo -s
[root@centos7 opc]# ip route show table t1
default via 10.1.1.1 dev ens3 
10.1.0.0/16 dev ens3 scope link src 10.1.1.12 

[root@centos7 opc]# ip route show table t2
default via 10.1.1.1 dev ens9 
10.1.0.0/16 dev ens9 scope link src 10.1.1.13 

[root@centos7 opc]# ip route show
10.1.1.0/24 dev ens3 proto kernel scope link src 10.1.1.12 
10.1.1.0/24 dev ens9 proto kernel scope link src 10.1.1.13 
169.254.0.0/16 dev ens3 scope link metric 1002 
169.254.0.0/16 dev ens9 scope link metric 1003 

[root@centos7 opc]# ping -I 10.1.1.12 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.1.1.12 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=61 time=10.6 ms

[root@centos7 opc]# ping -I 10.1.1.13 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.1.1.13 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=61 time=10.5 ms

Ping form hypervisor to VM IP’s works now.

# ping 10.1.1.12
PING 10.1.1.12 (10.1.1.12) 56(84) bytes of data.
64 bytes from 10.1.1.12: icmp_seq=1 ttl=64 time=0.223 ms

# ping 10.1.1.13
PING 10.1.1.13 (10.1.1.13) 56(84) bytes of data.
64 bytes from 10.1.1.13: icmp_seq=1 ttl=64 time=0.189 ms

Linux MSSQL Client

Quick note on connecting to a MSSQL database from Linux using tsql from FreeTDS. FreeTDS is a set of libraries for Unix and Linux that allows your programs to natively talk to Microsoft SQL Server and Sybase databases.

$ tsql -S DEVSQL1 -U <user> -P <password>
locale is "en_US.UTF-8"
locale charset is "UTF-8"
using default charset "UTF-8"
1> select @@version
2> go

Microsoft SQL Server 2016 (RTM-CU1) (KB3164674) - 13.0.2149.0 (X64) 
	Jul 11 2016 22:05:22 
	Copyright (c) Microsoft Corporation
	Developer Edition (64-bit) on Windows Server 2012 R2 Standard 6.3 <X64> (Build 9600: ) (Hypervisor)

(1 row affected)

1> use DEVDB1
2> go
1> SELECT * FROM INFORMATION_SCHEMA.TABLES 
2> go
TABLE_CATALOG	TABLE_SCHEMA	TABLE_NAME	TABLE_TYPE
DEVDB1	        TABLE_1  	JPS_DN	        BASE TABLE

Ref: https://tryolabs.com/blog/2012/06/25/connecting-sql-server-database-python-under-ubuntu/

Linux Mount nfsv4.2

Just a quick test on using nfs v4.2. This test was on a Ubuntu 17.4 server as well as client.

# cat /etc/exports 
<snip>
/DATA	*(ro,sync,no_root_squash,insecure)
/home   192.168.1.43(rw,insecure)

# systemctl restart nfs-kernel-server

# more /proc/fs/nfsd/versions 
+2 +3 +4 +4.1 +4.2

# mount -t nfs -o minorversion=2 server1:/DATA /DATA
# nfsstat -m
/mnt/home from server1:/home
 Flags:	rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.42,mountvers=3,mountport=41341,mountproto=udp,local_lock=none,addr=192.168.1.42

/DATA from server1:/DATA
 Flags:	rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=10.0.3.15,local_lock=none,addr=192.168.1.42

# rsync -a --progress ubuntu-17.04-desktop-amd64.iso /DATA/DATABANK/iso/
sending incremental file list
ubuntu-17.04-desktop-amd64.iso
  1,609,039,872 100%  157.75MB/s    0:00:09 (xfr#1, to-chk=0/1)

Linux for SPARC Boot Issue

I am running an Oracle Linux for SPARC ldom and had a couple boot issues recently. This may help getting past boot issues.

First issue was because I had a cdrom attached to the ldom and that path was not valid. Like unmounted NFS path for example. That caused a kernel dump and per the development list they will take this as a bug and fix in a future update. The workaround of course was simple once I figured out what was choking. Just remove invalid disk attachment.

The second issue was much more tricky and not until I spotted some selinux message on kernel panics did I realize some recent change I made with selinux profiles must have caused a missing config. The fix is to disable selinux but that was not as easy as I thought. Here is what I did and it may help someone else trying to pass kernel bootup parameters.
1. disable auto-boot

# ldm set-var auto-boot\?=false linuxsparc_ldom

2. Get into the boot prompt. This was the tricky part because the linux kernel started booting as soon as I type boot or boot disk. Plus either I did not have enough time before Silo boots the kernel or it is having an issue with normal Esc or Shift keystrokes to pause bootup. I am not sure but it kept booting whatever keystrokes I tried. What I ended up doing is using “-s”. I did “boot -s” which in normal SPARC world means it will boot the kernel in single user mode. I did not really expect openboot to pass single user to linux kernel boot but at least it stops then at boot prompt.

{0} ok boot disk -s
Boot device: /virtual-devices@100/channel-devices@200/disk@0  File and args: -s
SILO Version 1.4.14 - Rel: 4.0.18.el6
\
Welcome to Linux for SPARC!
Hit <TAB> for boot options
Your imagename `-s' and arguments `' have either wrong syntax,
or describe a label which is not present in silo.conf
Type `help' at the boot: prompt if you need it and then try again.
boot: 
4.1.12-32.el6uek.sparc64  linux-uek                

3. Now boot the linux kernel with selinux=0

boot: 4.1.12-32.el6uek.sparc64 selinux=0
Allocated 64 Megs of memory at 0x40000000 for kernel
Loaded kernel version 4.1.12
Loading initial ramdisk (25972306 bytes at 0

SSH Connection Manager

I previously wrote a quick post on using a connection manager in Linux. Link here:

Linux tabbed SSH connection manager

I have used for the most part something called the Gnome Connection Manager. However it is poorly maintained and had a few small annoyances also.

I revisited a utility called PAC Manager (link here https://sourceforge.net/projects/pacmanager/).

So far it does pretty much everything I need as far as maintaining details for server names and SSH login information. It does have tabbed windows, organize in groups and an amazing number of customization features. It also integrates pretty nicely with KeePass to maintain passwords with.

It would be better if the main distros include this tool but it does at least have .deb and .rpm packages.

I also gave a current version of Remmina another try as it seems best maintained of the bunch but it still gave me unexpected behavior. Like a SSH window just disappearing etc.

Migrating Ubuntu On a ZFS Root File System

I have written a couple articles about this here http://blog.ls-al.com/ubuntu-on-a-zfs-root-file-system-for-ubuntu-15-04/ and here http://blog.ls-al.com/ubuntu-on-a-zfs-root-file-system-for-ubuntu-14-04/

This is a quick update. After using virtualbox to export and import on a new machine my guest did not boot up all the way. I suspect I was just not seeing the message about manual/skip check of a file system and that the fstab entry for sda1 changed. Here is what I did. On bootup try “S” for skip if you are stuck. In my case I was stuck after a message about enabling encryption devices or something to that effect.

Check fstab and note disk device name.

root@ubuntu:~# cat /etc/fstab
/dev/disk/by-id/ata-VBOX_HARDDISK_VB7e932a52-ef3c41b0-part1 /boot/grub auto defaults 0 1 

Check if this device exists.

root@ubuntu:~# ls -l /dev/disk/by-id/ata-VBOX_HARDDISK_VB7e932a52-ef3c41b0*
ls: cannot access /dev/disk/by-id/ata-VBOX_HARDDISK_VB7e932a52-ef3c41b0*: No such file or directory

What is the correct device name.

root@ubuntu:~# ls -l /dev/disk/by-id/ata-VBOX_HARDDISK*                    
lrwxrwxrwx 1 root root  9 May  9 15:38 /dev/disk/by-id/ata-VBOX_HARDDISK_VBb0249023-5afef528 -> ../../sda
lrwxrwxrwx 1 root root 10 May  9 15:38 /dev/disk/by-id/ata-VBOX_HARDDISK_VBb0249023-5afef528-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 May  9 15:38 /dev/disk/by-id/ata-VBOX_HARDDISK_VBb0249023-5afef528-part2 -> ../../sda2

Keep old fstab and update with correct name.

root@ubuntu:~# cp /etc/fstab /root

root@ubuntu:~# vi /etc/fstab
root@ubuntu:~# sync
root@ubuntu:~# diff /etc/fstab /root/fstab 
1c1
< /dev/disk/by-id/ata-VBOX_HARDDISK_VBb0249023-5afef528-part1 /boot/grub auto defaults 0 1 
---
> /dev/disk/by-id/ata-VBOX_HARDDISK_VB7e932a52-ef3c41b0-part1 /boot/grub auto defaults 0 1 

Try rebooting now.

Nagios on Linux for SPARC

I recently experimented a little with Linux for SPARC(more here https://oss.oracle.com/projects/linux-sparc/) and found it to be surprisingly stable. One of the environments I support is a pure OVM for SPARC environment and no luxury of Linux. So I am running some open source tools like Nagios, HAproxy etc on Solaris. Nagios has worked ok but is painful to compile. There are also some bugs that cause high utilization.

I tried a Linux for SPARC instance and since they are pretty much like RedHat/Oracle/CentOS it means a fair bit of packages already exist. Nagios does not exist so I compiled it. Suffice to say installing dependencies from YUM and compiling was a breeze compared to Solaris.

You can pretty much follow this doc to the letter:
https://assets.nagios.com/downloads/nagioscore/docs/Installing_Nagios_Core_From_Source.pdf

Things to note.
1. By default the firewall does not allow inbound http.

2. If you have permission issues in the web frontend or something like Internal server error you can disable(quick test) and then configure selinux for nagios scripts.

# setenforce 0
# chcon -R -t httpd_sys_content_t /usr/local/nagios

3. Redo plugins with openssl for https checks. I wanted to do https checks.

# yum install openssl-devel
# pwd
/usr/src/nagios/nagios-plugins-2.1.1

# ./configure --with-openssl --with-nagios-user=nagios --with-nagios-group=nagios
[..]
                    --with-openssl: yes
# make
# make install

# /usr/local/nagios/libexec/check_http -H 10.2.10.33 -S -p 215 
HTTP OK: HTTP/1.1 200 OK - 2113 bytes in 0.017 second response time |time=0.016925s;;;0.000000 size=2113B;;;0

I made a https command as follow.

command.cfg
# 'check_https' command definition
define command{
        command_name    check_https
        command_line    $USER1$/check_http -H $HOSTADDRESS$ -S -p $ARG1$
        }

And referenced as follow.

storage.cfg
define service{
        use                             remote-service         ; Name of service template to use
        host_name                       zfssa1
        service_description             HTTPS
        check_command                   check_https!215
        notifications_enabled           0
        }

Linux tabbed SSH connection manager

I like to work in a tabbed SSH connection manager. Especially when I have hundred’s or thousands of machines to connect to. A connection manager like putty keeps track of machine names and login info. Using a tabbed interface like MTPutty can make your life a whole lot easier with treeview/groups and side by side terminals. And additionally if you can cluster the terminal commands it can be an added bonus.

So far I have not really liked anything in the Linux world as far as a SSH connection manager. Ubuntu does come with putty which seems to work the same as in the Windows world. Best I could find is an application written in Python called Gnome Connection Manager (gcm). In the Ubuntu 15.10 repos the package is called gnome-connection-manager. Be warned of a few things:

1. Seems the code is at least a few years old and the website does not seem to have documentation or any kind of discussion. The code is all python so you can look at fixing and emailing the owner.
2. Be sure to check the paste-right-click in ~/.gcm/gcm.conf. This can be a nasty setting if you did not expect it. I copy and paste a lot between a Windows desktop and a Linux guest and almost accidentally pasted garbage into a critical device.
3. Also check auto-copy-selection if you like putty style behavior where anything selected in your SSH terminal should be in the copy buffer.

I think if the developer put a little bit more love into gnome-connection-manager it would definitely be a keeper and first rate gnome app. I have looked at some other options like hotssh but worth checking out is Remmina. Unfortunately for me Remmina was very buggy.

Network Manager VPN Connections

I have documented previously that the Linux network manager can be used to connect to several different VPN gateways.  There are several network manager plugins available for the different VPN solutions.  The pptp plugin is used frequently but for newer Cisco gateways you should use the network-manager-openconnect-gnome plugin. You should use the network-manager-vpnc plugin to connect to older Cisco gateways.

The vpnc plugin also happens to work for Palo Alto GlobalProtect concentrators.  For the vpnc plugin to work with Palo Alto GlobalProtect gateways you need to:

– Enable X-Auth on your VPN gateway. You will also need the group name and password from the VPN administrator.

– Create a “Cisco compatible” VPN when creating your network manager connection.