pfsense 2.3 upgrade on Alix

I have been running pfsense on an Alix tiny computer for a long time. Pretty much have not touched it for years apart from occasional firewall rule change and pfsense auto upgrades. Recently I wanted to upgrade to pfsense 2.3 and had nothing but trouble. I am still not sure if this is a problem with the Alix specs, bad compact flash, unclean power down or the pfsense upgrade procedure.

I document here what I found but note I ended up backing up the configuration, flashing the same 4G compact flash and restoring configuration. So far it is working for me with the caveat I am still looking why the web interface bombed a few times. I think it is because you need to disable the updates available auto check on the System Info page but not sure. This issue did not affect the firewall functions so not a big deal for me right now.

Symptom
The upgrade will run for a very long time after downloading and stating upgrade started. When I say long I mean long. Finally I would get to a point like below.

Apr 13 19:16:04	php: config.inc: New alert found: Something went wrong when trying to update the fstab entry. Aborting upgrade.

You can check the full upgrade log under Diagnostics -> NanoBSD. Also worth nothing my first upgrade left me in a not booting state. I fished out a null modem cable and on the serial console pressed some keys and after that I could get back to previous version.

For reference here are some log snippets:

[2.2.4-RELEASE][admin@fw.local.domain]
Broadcast Message from root@fw.local.domain                              
        (no tty) at 8:39 CDT...                                                
NanoBSD Firmware upgrade in progress...                                        
                                                                            
Installing /root/latest.tgz.                  
NanoBSD upgrade starting

[..]

Installing /root/latest.tgz.
SLICE         2
OLDSLICE      1
TOFLASH       ada0s2
COMPLETE_PATH ada0s2a
GLABEL_SLICE  pfSense1
Wed Apr 13 19:08:58 CDT 2016

[..]

dd if=/dev/zero of=/dev/ada0s2 bs=1m count=1
1+0 records in
1+0 records out
1048576 bytes transferred in 0.257870 secs (4066298 bytes/sec)

/usr/bin/gzip -dc /root/latest.tgz | /bin/dd of=/dev/ada0s2 obs=64k
1890945+0 records in
14773+1 records out
968163840 bytes transferred in 311.756151 secs (3105516 bytes/sec)
After upgrade fdisk/bsdlabel

This looks to me like things starting to go wrong should not be unclean file system.

/sbin/fsck_ufs -y /dev/ada0s2a
** /dev/ada0s2a
** Last Mounted on /builder/pfSense-230/tmp/pfSense/_.mnt
** Phase 1 - Check Blocks and Sizes

CANNOT READ BLK: 1485136
CONTINUE? yes

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 1485136,
PARTIALLY TRUNCATED INODE I=93161
SALVAGE? yes

[..]

15850 files, 866146 used, 993212 free (1932 frags, 123910 blocks, 0.1% fragmentation)

***** FILE SYSTEM MARKED DIRTY *****

***** FILE SYSTEM WAS MODIFIED *****

***** PLEASE RERUN FSCK *****

/sbin/tunefs -L pfSense1 /dev/ada0s2a
Checking for post_upgrade_command...

[..]

fdisk: invalid fdisk partition table found
bsdlabel: /dev/ada0s3: no valid label found
bsdlabel: /dev/ada0s3: no valid label found
mount: /dev/ufs/pfSense1: R/W mount of /builder/pfSense-230/tmp/pfSense/_.mnt denied. Filesystem is not clean - run fsck.: Operation not permitted
cp: /tmp/pfSense1/etc/fstab: No such file or directory
sed: /tmp/pfSense1/etc/fstab: No such file or directory
umount: /tmp/pfSense1: not a file system root directory

fdisk/bsdlabel log:

Just a few references below.

Alix board specs here.
http://www.pcengines.ch/alix2d3.htm

For pfsense 2.3 it sounds like Alix bios need to be 99f.
https://doc.pfsense.org/index.php/ALIX_BIOS_Update_Procedure

This user describes a similar issue and re-image worked.
https://forum.pfsense.org/index.php?topic=71760.0

General.
https://doc.pfsense.org/index.php/Installing_pfSense
https://doc.pfsense.org/index.php/Writing_Disk_Images
Installing pfSense on a Compact Flash card
https://www.get-virtual.net/2014/09/16/build-firewall-appliance/