admin « Riaan's SysAdmin Blog

Author Archive

Jan 10

AWS Cognito and S3 Useful Commands

While I am delving into AWS Cognito and learning how it interacts with other services for example S3 object storage, I am jotting down some of the more useful CLI commands. This can be quite daunting to learn so it is very helpful to retain the commands for future reference. Of course this can all be done in the console also if that is your preference. I like the CLI (or even better would be Terraform or CloudFormation).

The examples may be useful when creating the authentication and authorization bits for a JavaScript SDK or Javascript framework (like Angular) application to upload files into a S3 bucket after being authenticated by the application. Note I use jq to filter output in many cases.

S3 Bucket

$ aws s3api create-bucket --bucket vault.mydomain.com --region us-east-1
$ aws s3api put-bucket-cors --bucket vault.mydomain.com --cors-configuration file://vault.mydomain.com-cors-policy.json
$ aws iam create-policy --policy-name vault.mydomain.com-admin-policy --policy-document file://vault.mydomain.com-admin-policy.json

Cognito User Pool

$ aws cognito-idp create-user-pool --pool-name mydomain-vault-user-pool   
$ aws cognito-idp list-user-pools --max-results 10 | jq -r '.UserPools[] | [.Id,.Name] | @csv' | grep vault  # get user-pool-id for create-user-pool-client step
$ aws cognito-idp create-user-pool-client --user-pool-id <your-userPoolId> --client-name mydomain-vault

Cognito Create an Admin User in the User Pool and do password reset flow

$ aws cognito-idp list-user-pool-clients –user-pool-id <your-userPoolId> --max-results 10 | jq -r '.UserPoolClients[] | [.ClientId,.ClientName] | @csv'  ## get client-id for next step
$ aws cognito-idp update-user-pool-client –user-pool-id <your-userPoolId> --client-id <your-clientId> --explicit-auth-flows ADMIN_NO_SRP_AUTH
$ aws cognito-idp admin-create-user –user-pool-id <your-userPoolId> --username admin --desired-delivery-mediums EMAIL --user-attributes Name=email,Value=admin@mydomain.com
## Check the above email address for temp password
$ aws cognito-idp admin-initiate-auth –user-pool-id <your-userPoolId> --client-id <your-clientId> --auth-flow ADMIN_NO_SRP_AUTH --auth-parameters USERNAME=admin,PASSWORD="<temp password from email"
## Use Very Long Session String from above output to respond and Set admin user password to <new password and complying to password policy>
$ aws cognito-idp admin-respond-to-auth-challenge –user-pool-id <your-userPoolId> --client-id <your-clientId> --challenge-name NEW_PASSWORD_REQUIRED --challenge-responses NEW_PASSWORD="<your-new-password>",USERNAME=admin --session "<very long session string>"
$ aws cognito-idp update-user-pool-client –user-pool-id <your-userPoolId> --client-id <your-clientId> –explicit-auth-flows

Cognito Create Identity Pool

$ aws cognito-idp describe-user-pool –user-pool-id <your-userPoolId> | jq -r '.[] | [.Name,.Arn] | @csv' 	## get UserPool Arn
$ aws cognito-identity create-identity-pool --identity-pool-name "mydomain vault identity pool" --allow-unauthenticated-identities --cognito-identity-providers ProviderName="cognito-idp.us-east-1.amazonaws.com/<your-userPoolId>",ClientId="<your-clientId>"
$ aws iam create-role --role-name vault.mydomain.com-admin-role --assume-role-policy-document file://vault.mydomain.com-admin-trust-role.json
$ aws cognito-identity list-identity-pools --max-results 3 | jq -r '.IdentityPools[] | [.IdentityPoolId,.IdentityPoolName] | @csv' | grep vault ## get identity pool id
## use our new role for authenticated role. for unauthenticated I used an old one since I don't plan unauthenticated access here. If you do need unauthenticated create a role and use below.
$ aws cognito-identity set-identity-pool-roles --identity-pool-id <your-identityPoolId> --roles authenticated="<your-arn-authenticated-role>",unauthenticated="<your-arn-unauthenticated-role>"

In the console change Authenticated role selection to "Choose role from token" and Role resolution "Use default Authenticated role". See if this can be done from CLI.

IAM Attach Role to Policy

## get role names just for your verification
$ aws cognito-identity get-identity-pool-roles –identity-pool-id <your-identityPoolId> | jq -r '[.IdentityPoolId,.Roles.authenticated,.Roles.unauthenticated] | @csv' 
$ aws iam list-policies | jq -r '.Policies[] | [.PolicyName,.Arn] | @csv' | grep vault	## get policy Arn
$ aws iam attach-role-policy --policy-arn arn:aws:iam::660032875792:policy/vault.mydomain.com-admin-policy --role-name vault.mydomain.com-admin-role

Application

Application need to use correct UserPoolId, App ClientId, identityPoolId, S3 bucket name, region. Very important is to understand "Integrating a User Pool with an Identity Pool". Example: https://docs.aws.amazon.com/cognito/latest/developerguide/amazon-cognito-integrating-user-pools-with-identity-pools.html

Appendix A: JSON Source used in above commands

$ cat mydomain-vault-s3-upload/vault.mydomain.com-admin-policy.json 
{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "s3:*"
 ],
 "Resource": [
        "arn:aws:s3:::vault.mydomain.com",
 "arn:aws:s3:::vault.mydomain.com/*"
 ]
 }
 ]
}
$ cat mydomain-vault-s3-upload/vault.mydomain.com-admin-trust-role.json 
{
 "Version": "2012-10-17",
 "Statement": {
 "Effect": "Allow",
"Principal": {
 "Federated": "cognito-identity.amazonaws.com"
 },
 "Action": "sts:AssumeRoleWithWebIdentity",
 "Condition": {
 "StringEquals": {
 "cognito-identity.amazonaws.com:aud": "<your-identityPoolId>"
 }
 }
 }
}
$ cat mydomain-vault-s3-upload/vault.mydomain.com-cors-policy.json 
{
 "CORSRules": [
 {
 "AllowedOrigins": ["*"],
 "AllowedHeaders": ["*"],
 "AllowedMethods": ["PUT", "GET", "POST", "DELETE"],
 "MaxAgeSeconds": 3000,
 "ExposeHeaders": ["ETag"]
 }

Comments Off on AWS Cognito and S3 Useful Commands
comments

Posted in Uncategorized

Jan 04

Object Storage Listing with Rclone and jq

Some examples of using rclone and jq to see object listing in a bucket. These examples was using Oracle (OCI) Object Storage but since this is rclone it should not matter what the target is.

Rclone retrieving JSON object listing of a bucket:

$ rclone lsjson -R s3_ashburn:APPS 
[
{"Path":"config","Name":"config","Size":155,"MimeType":"application/octet-stream","ModTime":"2018-11-02T18:01:31.028653533Z","IsDir":false},
{"Path":"data","Name":"data","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533157179Z","IsDir":true},
{"Path":"index","Name":"index","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533226556Z","IsDir":true},
{"Path":"keys","Name":"keys","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533246534Z","IsDir":true},
{"Path":"snapshots","Name":"snapshots","Size":0,"MimeType":"inode/directory","ModTime":"2019-01-04T15:31:54.533266804Z","IsDir":true},
{"Path":"index/6f0870dc3d699c0e550f62c535f11a3e52396f45d9c3439760a5f648ee2f1533","Name":"6f0870dc3d699c0e550f62c535f11a3e52396f45d9c3439760a5f648ee2f1533","Size":37828
350,"MimeType":"application/octet-stream","ModTime":"2019-01-03T21:27:05Z","IsDir":false},
{"Path":"index/b20a6e07f25d834739e3c3fd82cf3b7ade3e7f1f0f286aab61006532621220ae","Name":"b20a6e07f25d834739e3c3fd82cf3b7ade3e7f1f0f286aab61006532621220ae","Size":36726
493,"MimeType":"application/octet-stream","ModTime":"2019-01-03T21:27:02Z","IsDir":false},

Use jq select to grab older than certain dates:

$ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | select (."ModTime" < "2018-12-01")|.Name'
ffea09b644533ddcde68a93095bc512646fd0ac0557d39e6e06e004bf73b6bed
ffef7980ade85ea2d9b436c40df46384bbbe8e7e6e71219aff0757ad90f1652f
fff3f56e384ab055c3aa4b6e2dd527c368bf2280863d357e577402460fe9d41a

Use jq csv filter and specific fields

$ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | [.Name,.Size] | @csv'

Use jq select for older than certain date, specific fields and csv

$ rclone lsjson -R s3_ashburn:APPS | jq -r '.[] | select (."ModTime" < "2018-11-01") | [.Name,.Size,.ModTime] | @csv'

Rclone size

$ rclone size s3_ashburn:APPS --json 
{"count":8088,"bytes":38670955795}

Rclone size and jq csv filter

$ rclone size s3_ashburn:APPS --json | jq -r '[.count,.bytes] | @csv'
8088,38670955795

Comments Off on Object Storage Listing with Rclone and jq
comments

Posted in rclone

Jan 02

Vim And Auto Indent Unwanted New Lines

Recently I experienced a very annoying issue with copy/paste. Pasting a line using the Gnome terminal Control-V or Right Mouse Button caused the line to insert with an unwanted new line. I am not sure when or why this started but it could have to do with the file being called "changelog" and/or vim identifying the file as type "changelog". Using ":set" while editing the file does list autoindent.

Note that this issue does not appear with yank and put (Shift-Y and Shift-P) commands. For the moment I am getting around this by adding ":set pastetoggle=<F5>" to my .vimrc. Then before pasting press F5. You could also just do ":set paste" while editing and the paste would work normal for that paste.

Comments Off on Vim And Auto Indent Unwanted New Lines
comments

Posted in Vim

Dec 07

Solaris Boot Environment And Lost Changes

Maybe you have similarly been perplexed like me when you are missing a file after a new boot environment(BE) was activated.

In my example I know I created a file in my /root/Desktop folder and after pkg update and the resulting new BE was automatically activated and I rebooted I noticed my file was missing.

This can be quite nasty if you have made system changes and not noticed they all were lost. For example updating /etc/hosts etc…

Sequence that is faulty

After OS initial installation and before updates (pkg update)
Save a text file /root/Desktop/20181206-pkg-update
Current BE is openindiana

Do updates and reboot
Current BE is openindiana-1
File is gone

Sequence that works

Save a text file /root/Desktop/20181206-pkg-update
beadm create new BE and activate
reboot immediately
pkg update

Question:

Is there an easier way to update to avoid this problem? Maybe passing parms?

Read here also....
https://blog.teodeh.com/2013/02/25/solaris_11_patching_and_updates/

Tip: If just restoring a file you can just use beadm mount to find a specific file for retrieval or comparison with existing files.

Comments Off on Solaris Boot Environment And Lost Changes
comments

Posted in Solaris

Nov 10

Restic and Oracle OCI Object Storage

It seems that after some time went by the S3 compatible object storage OCI interface can now work with restic directly and not necessary to use rclone. Tests a few months ago this did not work.

Using S3 directly mean we may not have this issue we see when using restic + rclone:
rclone: 2018/11/02 20:04:16 ERROR : data/fa/fadbb4f1d9172a4ecb591ddf5677b0889c16a8b98e5e3329d63aa152e235602e: Didn't finish writing GET request (wrote 9086/15280 bytes): http2: stream closed

This shows how I setup restic to Oracle OCI object storage(no rclone required).

Current restic env pointing to rclone.conf
##########################################

# more /root/.restic-env 
export RESTIC_REPOSITORY="rclone:s3_servers_ashburn:bucket1"
export RESTIC_PASSWORD="blahblah"

# more /root/.config/rclone/rclone.conf 
[s3_servers_phoenix]
type = s3
env_auth = false
access_key_id =  
secret_access_key =  
region = us-phoenix-1
endpoint = <client-id>.compat.objectstorage.us-phoenix-1.oraclecloud.com
location_constraint = 
acl = private
server_side_encryption = 
storage_class = 
[s3_servers_ashburn]
type = s3
env_auth = false
access_key_id =  
secret_access_key = 
region = us-ashburn-1
endpoint = <client-id>.compat.objectstorage.us-ashburn-1.oraclecloud.com
location_constraint =
acl = private
server_side_encryption =

New restic env pointing to S3 style
###################################

# more /root/.restic-env 
export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export RESTIC_REPOSITORY="s3:<client-id>.compat.objectstorage.us-ashburn-1.oraclecloud.com/bucket1"
export RESTIC_PASSWORD="blahblah"

# . /root/.restic-env

# /usr/local/bin/restic snapshots
repository 26e5f447 opened successfully, password is correct
ID        Date                 Host             Tags        Directory
----------------------------------------------------------------------
dc9827fd  2018-08-31 21:20:02  server1                      /etc
cb311517  2018-08-31 21:20:04  server1                      /home
f65a3bb5  2018-08-31 21:20:06  server1                      /var
{...}
----------------------------------------------------------------------
36 snapshots

Comments Off on Restic and Oracle OCI Object Storage
comments

Posted in OCI, rclone, restic

Oct 10

AWS Lambda and Python

AWS Lambda is a server less computing platform. You can execute your code without provisioning or managing servers.

Tested an example of copying a text file dropped into one S3 bucket to another.

1. Create a IAM role with the CloudWatch and S3 policies.
Call role lambda_s3 and add policies: AWSOpsWorksCloudWatchLogs, AmazonS3FullAccess
2. Create two S3 buckets for source and target.
3. Create Lambda function for Copying a file from one bucket to another.
Author from scratch, Name = copyS3toS3 Python2.7, Existing Role = lambda_s3
Add S3 from left selections
Trigger select the source bucket, Object Created(All), Suffix = .txt, Check Enable Trigger
Click on function copyS3toS3 and add python code as showed in Appendix A
4. Save the Lambda function and upload a text file to the source s3 bucket to test.
5. You can go to Cloudwatch logs to root cause if test .txt file not showing up in target.

Appendix A: Lambda function python code
#######################################

from __future__ import print_function

import json
import boto3
import time
import urllib

print('Loading function')

s3 = boto3.client("s3")

def lambda_handler(event,context):
  source_bucket = event['Records'][0]['s3']['bucket']['name']
  key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key'])
  target_bucket = 'iqonda-test02'  # target s3 bucket name
  copy_source = {'Bucket':source_bucket, 'Key':key}
  
  try:
    print('Waiting for the file persist in the source bucket')
    waiter = s3.get_waiter('object_exists')
    waiter.wait(Bucket=source_bucket, Key=key)
    print('Copying object from source s3 bucket to target s3 bucket')
    s3.copy_object(Bucket=target_bucket, Key=key, CopySource=copy_source)
  except Exception as e:
    print(e)
    print('Error getting object {} from bucket {}. Make sure they exist '
              'and your bucket is in the same region as this '
              'function.'.format(key, bucket))
    raise e

Appendix B:
###########
https://gist.github.com/anonymous/0f6b21d1586bd291d4ad0cc84c6383bb#file-s3-devnull-py

Comments Off on AWS Lambda and Python
comments

Posted in AWS

Sep 15

Terraform with Azure

Azure provides a cloud shell with Terraform already installed and even an editor aware of Terraform source. I tested creating a network and VM using the cloud shell in the browser.

REF: https://docs.microsoft.com/en-us/azure/virtual-machines/linux/terraform-install-configure

Note: I am not showing dry run (terraform plan) but you should always be in the habit of plan before apply.

###################################################################
Using cloud shell prompts for a storage account tied to your shell.
###################################################################

Your cloud drive has been created in:

Subscription Id: b[..]b
Resource group:  cloud-shell-storage-southcentralus
Storage account: c[..]4
File share:      cs-[..]2

Initializing your account for Cloud Shell...\
Requesting a Cloud Shell.Succeeded.
Connecting terminal...

Welcome to Azure Cloud Shell

Type "az" to use Azure CLI 2.0
Type "help" to learn about Cloud Shell

#######
Test az
#######

riaan@Azure:~$ az vm list
[
  {
    "availabilitySet": null,
    "diagnosticsProfile": {
      "bootDiagnostics": {
        "enabled": true,
        "storageUri": "https://sitesdiag.blob.core.windows.net/"
      }
    },
    "hardwareProfile": {
      "vmSize": "Standard_B1s"
    },
 [..]
  }
]

###################################################################################################################
Check folder and space available
###################################################################################################################

riaan@Azure:~$ ls
clouddrive
riaan@Azure:~$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      5.0G   11M  4.7G   1% /home/riaan

riaan@Azure:~$ az account show --query "{subscriptionId:id, tenantId:tenantId}"
{
  "subscriptionId": "b[..]",
  "tenantId": "f[..]"
}

###################################################################################################################
Simple terraform test add a resource group
###################################################################################################################

riaan@Azure:~$ mkdir sites
riaan@Azure:~$ cd sites
riaan@Azure:~/sites$ vi test.tf
riaan@Azure:~/sites$ terraform init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "azurerm" (1.15.0)...
[..]
* provider.azurerm: version = "~> 1.15"
Terraform has been successfully initialized!

riaan@Azure:~/sites$ terraform apply
[..]
  + create

Terraform will perform the following actions:

  + azurerm_resource_group.rg
      id:       <computed>
      location: "centralus"
      name:     "testResourceGroup"
      tags.%:   <computed>

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

azurerm_resource_group.rg: Creating...
  location: "" => "centralus"
  name:     "" => "testResourceGroup"
  tags.%:   "" => "<computed>"
azurerm_resource_group.rg: Creation complete after 0s (ID: /subscriptions/b[..]/resourceGroups/testResourceGroup)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

#######################################################################
Simple terraform test. Destroy a resource group. I just comment it out.
#######################################################################

riaan@Azure:~/sites$ vi test.tf
riaan@Azure:~/sites$ terraform apply
azurerm_resource_group.rg: Refreshing state... (ID: /subscriptions/b[..]/resourceGroups/testResourceGroup)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - azurerm_resource_group.rg

Plan: 0 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

azurerm_resource_group.rg: Destroying... (ID: /subscriptions/b[..]/resourceGroups/testResourceGroup)
azurerm_resource_group.rg: Still destroying... (ID: /subscriptions/b[..]/resourceGroups/testResourceGroup, 10s elapsed)
[..]
azurerm_resource_group.rg: Destruction complete after 45s

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

###################################################################################################################
Create VM plus network
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/terraform-create-complete-vm
###################################################################################################################

riaan@Azure:~/sites$ rm test.tf
riaan@Azure:~/sites$ vi main.tf
riaan@Azure:~/sites$ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + azurerm_network_interface.myterraformnic
      id:                                                                    <computed>
      applied_dns_servers.#:                                                 <computed>
      dns_servers.#:                                                         <computed>
      enable_accelerated_networking:                                         "false"
      enable_ip_forwarding:                                                  "false"
      internal_dns_name_label:                                               <computed>
      internal_fqdn:                                                         <computed>
      ip_configuration.#:                                                    "1"
      ip_configuration.0.application_gateway_backend_address_pools_ids.#:    <computed>
      ip_configuration.0.application_security_group_ids.#:                   <computed>
      ip_configuration.0.load_balancer_backend_address_pools_ids.#:          <computed>
      ip_configuration.0.load_balancer_inbound_nat_rules_ids.#:              <computed>
      ip_configuration.0.name:                                               "myNicConfiguration"
      ip_configuration.0.primary:                                            <computed>
      ip_configuration.0.private_ip_address_allocation:                      "dynamic"
      ip_configuration.0.public_ip_address_id:                               "${azurerm_public_ip.myterraformpublicip.id}"
      ip_configuration.0.subnet_id:                                          "${azurerm_subnet.myterraformsubnet.id}"
      location:                                                              "eastus"
      mac_address:                                                           <computed>
      name:                                                                  "myNIC"
      network_security_group_id:                                             "${azurerm_network_security_group.myterraformnsg.id}"
      private_ip_address:                                                    <computed>
      private_ip_addresses.#:                                                <computed>
      resource_group_name:                                                   "myResourceGroup"
      tags.%:                                                                "1"
      tags.environment:                                                      "Terraform Demo"
      virtual_machine_id:                                                    <computed>

  + azurerm_network_security_group.myterraformnsg
      id:                                                                    <computed>
      location:                                                              "eastus"
      name:                                                                  "myNetworkSecurityGroup"
      resource_group_name:                                                   "myResourceGroup"
      security_rule.#:                                                       "1"
      security_rule.2090309246.access:                                       "Allow"
      security_rule.2090309246.description:                                  ""
      security_rule.2090309246.destination_address_prefix:                   "*"
      security_rule.2090309246.destination_address_prefixes.#:               "0"
      security_rule.2090309246.destination_application_security_group_ids.#: "0"
      security_rule.2090309246.destination_port_range:                       "22"
      security_rule.2090309246.destination_port_ranges.#:                    "0"
      security_rule.2090309246.direction:                                    "Inbound"
      security_rule.2090309246.name:                                         "SSH"
      security_rule.2090309246.priority:                                     "1001"
      security_rule.2090309246.protocol:                                     "Tcp"
      security_rule.2090309246.source_address_prefix:                        "*"
      security_rule.2090309246.source_address_prefixes.#:                    "0"
      security_rule.2090309246.source_application_security_group_ids.#:      "0"
      security_rule.2090309246.source_port_range:                            "*"
      security_rule.2090309246.source_port_ranges.#:                         "0"
      tags.%:                                                                "1"
      tags.environment:                                                      "Terraform Demo"

  + azurerm_public_ip.myterraformpublicip
      id:                                                                    <computed>
      fqdn:                                                                  <computed>
      ip_address:                                                            <computed>
      location:                                                              "eastus"
      name:                                                                  "myPublicIP"
      public_ip_address_allocation:                                          "dynamic"
      resource_group_name:                                                   "myResourceGroup"
      sku:                                                                   "Basic"
      tags.%:                                                                "1"
      tags.environment:                                                      "Terraform Demo"

  + azurerm_resource_group.myterraformgroup
      id:                                                                    <computed>
      location:                                                              "eastus"
      name:                                                                  "myResourceGroup"
      tags.%:                                                                "1"
      tags.environment:                                                      "Terraform Demo"

  + azurerm_storage_account.mystorageaccount
      id:                                                                    <computed>
      access_tier:                                                           <computed>
      account_encryption_source:                                             "Microsoft.Storage"
      account_kind:                                                          "Storage"
      account_replication_type:                                              "LRS"
      account_tier:                                                          "Standard"
      enable_blob_encryption:                                                "true"
      enable_file_encryption:                                                "true"
      identity.#:                                                            <computed>
      location:                                                              "eastus"
      name:                                                                  "diag${random_id.randomId.hex}"
      primary_access_key:                                                    <computed>
      primary_blob_connection_string:                                        <computed>
      primary_blob_endpoint:                                                 <computed>
      primary_connection_string:                                             <computed>
      primary_file_endpoint:                                                 <computed>
      primary_location:                                                      <computed>
      primary_queue_endpoint:                                                <computed>
      primary_table_endpoint:                                                <computed>
      resource_group_name:                                                   "myResourceGroup"
      secondary_access_key:                                                  <computed>
      secondary_blob_connection_string:                                      <computed>
      secondary_blob_endpoint:                                               <computed>
      secondary_connection_string:                                           <computed>
      secondary_location:                                                    <computed>
      secondary_queue_endpoint:                                              <computed>
      secondary_table_endpoint:                                              <computed>
      tags.%:                                                                "1"
      tags.environment:                                                      "Terraform Demo"

  + azurerm_subnet.myterraformsubnet
      id:                                                                    <computed>
      address_prefix:                                                        "10.0.1.0/24"
      ip_configurations.#:                                                   <computed>
      name:                                                                  "mySubnet"
      resource_group_name:                                                   "myResourceGroup"
      virtual_network_name:                                                  "myVnet"

  + azurerm_virtual_machine.myterraformvm
      id:                                                                    <computed>
      availability_set_id:                                                   <computed>
      boot_diagnostics.#:                                                    "1"
      boot_diagnostics.0.enabled:                                            "true"
      boot_diagnostics.0.storage_uri:                                        "${azurerm_storage_account.mystorageaccount.primary_blob_endpoint}"
      delete_data_disks_on_termination:                                      "false"
      delete_os_disk_on_termination:                                         "false"
      identity.#:                                                            <computed>
      location:                                                              "eastus"
      name:                                                                  "myVM"
      network_interface_ids.#:                                               <computed>
      os_profile.#:                                                          "1"
      os_profile.1770182618.admin_password:                                  <sensitive>
      os_profile.1770182618.admin_username:                                  "azureuser"
      os_profile.1770182618.computer_name:                                   "myvm"
      os_profile.1770182618.custom_data:                                     <computed>
      os_profile_linux_config.#:                                             "1"
      os_profile_linux_config.69840937.disable_password_authentication:      "true"
      os_profile_linux_config.69840937.ssh_keys.#:                           "1"
      os_profile_linux_config.69840937.ssh_keys.0.key_data:                  "ssh-rsa AAAAB3Nz{snip}hwhqT9h"
      os_profile_linux_config.69840937.ssh_keys.0.path:                      "/home/azureuser/.ssh/authorized_keys"
      resource_group_name:                                                   "myResourceGroup"
      storage_data_disk.#:                                                   <computed>
      storage_image_reference.#:                                             "1"
      storage_image_reference.363552096.id:                                  ""
      storage_image_reference.363552096.offer:                               "UbuntuServer"
      storage_image_reference.363552096.publisher:                           "Canonical"
      storage_image_reference.363552096.sku:                                 "16.04.0-LTS"
      storage_image_reference.363552096.version:                             "latest"
      storage_os_disk.#:                                                     "1"
      storage_os_disk.0.caching:                                             "ReadWrite"
      storage_os_disk.0.create_option:                                       "FromImage"
      storage_os_disk.0.disk_size_gb:                                        <computed>
      storage_os_disk.0.managed_disk_id:                                     <computed>
      storage_os_disk.0.managed_disk_type:                                   "Premium_LRS"
      storage_os_disk.0.name:                                                "myOsDisk"
      storage_os_disk.0.write_accelerator_enabled:                           "false"
      tags.%:                                                                "1"
      tags.environment:                                                      "Terraform Demo"
      vm_size:                                                               "Standard_DS1_v2"

  + azurerm_virtual_network.myterraformnetwork
      id:                                                                    <computed>
      address_space.#:                                                       "1"
      address_space.0:                                                       "10.0.0.0/16"
      location:                                                              "eastus"
      name:                                                                  "myVnet"
      resource_group_name:                                                   "myResourceGroup"
      subnet.#:                                                              <computed>
      tags.%:                                                                "1"
      tags.environment:                                                      "Terraform Demo"

  + random_id.randomId
      id:                                                                    <computed>
      b64:                                                                   <computed>
      b64_std:                                                               <computed>
      b64_url:                                                               <computed>
      byte_length:                                                           "8"
      dec:                                                                   <computed>
      hex:                                                                   <computed>
      keepers.%:                                                             "1"
      keepers.resource_group:                                                "myResourceGroup"


Plan: 9 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

azurerm_resource_group.myterraformgroup: Creating...
  location:         "" => "eastus"
  name:             "" => "myResourceGroup"
  tags.%:           "" => "1"
  tags.environment: "" => "Terraform Demo"
azurerm_resource_group.myterraformgroup: Creation complete after 0s (ID: /subscriptions/b[..]/resourceGroups/myResourceGroup)
azurerm_virtual_network.myterraformnetwork: Creating...
  address_space.#:     "" => "1"
  address_space.0:     "" => "10.0.0.0/16"
  location:            "" => "eastus"
  name:                "" => "myVnet"
  resource_group_name: "" => "myResourceGroup"
  subnet.#:            "" => "<computed>"
  tags.%:              "" => "1"
  tags.environment:    "" => "Terraform Demo"
azurerm_public_ip.myterraformpublicip: Creating...
  fqdn:                         "" => "<computed>"
  ip_address:                   "" => "<computed>"
  location:                     "" => "eastus"
  name:                         "" => "myPublicIP"
  public_ip_address_allocation: "" => "dynamic"
  resource_group_name:          "" => "myResourceGroup"
  sku:                          "" => "Basic"
  tags.%:                       "" => "1"
  tags.environment:             "" => "Terraform Demo"
random_id.randomId: Creating...
  b64:                    "" => "<computed>"
  b64_std:                "" => "<computed>"
  b64_url:                "" => "<computed>"
  byte_length:            "" => "8"
  dec:                    "" => "<computed>"
  hex:                    "" => "<computed>"
  keepers.%:              "" => "1"
  keepers.resource_group: "" => "myResourceGroup"
random_id.randomId: Creation complete after 0s (ID: ZZic4bhOapM)
azurerm_network_security_group.myterraformnsg: Creating...
  location:                                                              "" => "eastus"
  name:                                                                  "" => "myNetworkSecurityGroup"
  resource_group_name:                                                   "" => "myResourceGroup"
  security_rule.#:                                                       "" => "1"
  security_rule.2090309246.access:                                       "" => "Allow"
  security_rule.2090309246.description:                                  "" => ""
  security_rule.2090309246.destination_address_prefix:                   "" => "*"
  security_rule.2090309246.destination_address_prefixes.#:               "" => "0"
  security_rule.2090309246.destination_application_security_group_ids.#: "" => "0"
  security_rule.2090309246.destination_port_range:                       "" => "22"
  security_rule.2090309246.destination_port_ranges.#:                    "" => "0"
  security_rule.2090309246.direction:                                    "" => "Inbound"
  security_rule.2090309246.name:                                         "" => "SSH"
  security_rule.2090309246.priority:                                     "" => "1001"
  security_rule.2090309246.protocol:                                     "" => "Tcp"
  security_rule.2090309246.source_address_prefix:                        "" => "*"
  security_rule.2090309246.source_address_prefixes.#:                    "" => "0"
  security_rule.2090309246.source_application_security_group_ids.#:      "" => "0"
  security_rule.2090309246.source_port_range:                            "" => "*"
  security_rule.2090309246.source_port_ranges.#:                         "" => "0"
  tags.%:                                                                "" => "1"
  tags.environment:                                                      "" => "Terraform Demo"
azurerm_storage_account.mystorageaccount: Creating...
  access_tier:                      "" => "<computed>"
  account_encryption_source:        "" => "Microsoft.Storage"
  account_kind:                     "" => "Storage"
  account_replication_type:         "" => "LRS"
  account_tier:                     "" => "Standard"
  enable_blob_encryption:           "" => "true"
  enable_file_encryption:           "" => "true"
  identity.#:                       "" => "<computed>"
  location:                         "" => "eastus"
  name:                             "" => "diag6[..]"
  primary_access_key:               "<sensitive>" => "<sensitive>"
  primary_blob_connection_string:   "<sensitive>" => "<sensitive>"
  primary_blob_endpoint:            "" => "<computed>"
  primary_connection_string:        "<sensitive>" => "<sensitive>"
  primary_file_endpoint:            "" => "<computed>"
  primary_location:                 "" => "<computed>"
  primary_queue_endpoint:           "" => "<computed>"
  primary_table_endpoint:           "" => "<computed>"
  resource_group_name:              "" => "myResourceGroup"
  secondary_access_key:             "<sensitive>" => "<sensitive>"
  secondary_blob_connection_string: "<sensitive>" => "<sensitive>"
  secondary_blob_endpoint:          "" => "<computed>"
  secondary_connection_string:      "<sensitive>" => "<sensitive>"
  secondary_location:               "" => "<computed>"
  secondary_queue_endpoint:         "" => "<computed>"
  secondary_table_endpoint:         "" => "<computed>"
  tags.%:                           "" => "1"
  tags.environment:                 "" => "Terraform Demo"
azurerm_public_ip.myterraformpublicip: Creation complete after 2s (ID: /subscriptions/b[..]-...t.Network/publicIPAddresses/myPublicIP)
azurerm_network_security_group.myterraformnsg: Creation complete after 2s (ID: /subscriptions/b[..]-...kSecurityGroups/myNetworkSecurityGroup)
azurerm_virtual_network.myterraformnetwork: Still creating... (10s elapsed)
azurerm_storage_account.mystorageaccount: Still creating... (10s elapsed)
azurerm_storage_account.mystorageaccount: Creation complete after 19s (ID: /subscriptions/b[..]-...e/storageAccounts/diag6[..])
azurerm_virtual_network.myterraformnetwork: Still creating... (20s elapsed)
azurerm_virtual_network.myterraformnetwork: Creation complete after 22s (ID: /subscriptions/b[..]-...crosoft.Network/virtualNetworks/myVnet)
azurerm_subnet.myterraformsubnet: Creating...
  address_prefix:       "" => "10.0.1.0/24"
  ip_configurations.#:  "" => "<computed>"
  name:                 "" => "mySubnet"
  resource_group_name:  "" => "myResourceGroup"
  virtual_network_name: "" => "myVnet"
azurerm_subnet.myterraformsubnet: Still creating... (10s elapsed)
azurerm_subnet.myterraformsubnet: Creation complete after 11s (ID: /subscriptions/b[..]-...irtualNetworks/myVnet/subnets/mySubnet)
azurerm_network_interface.myterraformnic: Creating...
  applied_dns_servers.#:                                              "" => "<computed>"
  dns_servers.#:                                                      "" => "<computed>"
  enable_accelerated_networking:                                      "" => "false"
  enable_ip_forwarding:                                               "" => "false"
  internal_dns_name_label:                                            "" => "<computed>"
  internal_fqdn:                                                      "" => "<computed>"
  ip_configuration.#:                                                 "" => "1"
  ip_configuration.0.application_gateway_backend_address_pools_ids.#: "" => "<computed>"
  ip_configuration.0.application_security_group_ids.#:                "" => "<computed>"
  ip_configuration.0.load_balancer_backend_address_pools_ids.#:       "" => "<computed>"
  ip_configuration.0.load_balancer_inbound_nat_rules_ids.#:           "" => "<computed>"
  ip_configuration.0.name:                                            "" => "myNicConfiguration"
  ip_configuration.0.primary:                                         "" => "<computed>"
  ip_configuration.0.private_ip_address_allocation:                   "" => "dynamic"
  ip_configuration.0.public_ip_address_id:                            "" => "/subscriptions/b[..]/resourceGroups/myResourceGroup/providers/Microsoft.Network/publicIPAddresses/myPublicIP"
  ip_configuration.0.subnet_id:                                       "" => "/subscriptions/b[..]/resourceGroups/myResourceGroup/providers/Microsoft.Network/virtualNetworks/myVnet/subnets/mySubnet"
  location:                                                           "" => "eastus"
  mac_address:                                                        "" => "<computed>"
  name:                                                               "" => "myNIC"
  network_security_group_id:                                          "" => "/subscriptions/b[..]/resourceGroups/myResourceGroup/providers/Microsoft.Network/networkSecurityGroups/myNetworkSecurityGroup"
  private_ip_address:                                                 "" => "<computed>"
  private_ip_addresses.#:                                             "" => "<computed>"
  resource_group_name:                                                "" => "myResourceGroup"
  tags.%:                                                             "" => "1"
  tags.environment:                                                   "" => "Terraform Demo"
  virtual_machine_id:                                                 "" => "<computed>"
azurerm_network_interface.myterraformnic: Creation complete after 1s (ID: /subscriptions/b[..]-...rosoft.Network/networkInterfaces/myNIC)
azurerm_virtual_machine.myterraformvm: Creating...
  availability_set_id:                                              "" => "<computed>"
  boot_diagnostics.#:                                               "" => "1"
  boot_diagnostics.0.enabled:                                       "" => "true"
  boot_diagnostics.0.storage_uri:                                   "" => "https://diag[..].blob.core.windows.net/"
  delete_data_disks_on_termination:                                 "" => "false"
  delete_os_disk_on_termination:                                    "" => "false"
  identity.#:                                                       "" => "<computed>"
  location:                                                         "" => "eastus"
  name:                                                             "" => "myVM"
  network_interface_ids.#:                                          "" => "1"
  network_interface_ids.0:                                          "" => "/subscriptions/b[..]/resourceGroups/myResourceGroup/providers/Microsoft.Network/networkInterfaces/myNIC"
  os_profile.#:                                                     "" => "1"
  os_profile.1770182618.admin_password:                             "<sensitive>" => "<sensitive>"
  os_profile.1770182618.admin_username:                             "" => "azureuser"
  os_profile.1770182618.computer_name:                              "" => "myvm"
  os_profile.1770182618.custom_data:                                "" => "<computed>"
  os_profile_linux_config.#:                                        "" => "1"
  os_profile_linux_config.69840937.disable_password_authentication: "" => "true"
  os_profile_linux_config.69840937.ssh_keys.#:                      "" => "1"
  os_profile_linux_config.69840937.ssh_keys.0.key_data:             "" => "ssh-rsa AAAAB3Nz{snip}hwhqT9h"
  os_profile_linux_config.69840937.ssh_keys.0.path:                 "" => "/home/azureuser/.ssh/authorized_keys"
  resource_group_name:                                              "" => "myResourceGroup"
  storage_data_disk.#:                                              "" => "<computed>"
  storage_image_reference.#:                                        "" => "1"
  storage_image_reference.363552096.id:                             "" => ""
  storage_image_reference.363552096.offer:                          "" => "UbuntuServer"
  storage_image_reference.363552096.publisher:                      "" => "Canonical"
  storage_image_reference.363552096.sku:                            "" => "16.04.0-LTS"
  storage_image_reference.363552096.version:                        "" => "latest"
  storage_os_disk.#:                                                "" => "1"
  storage_os_disk.0.caching:                                        "" => "ReadWrite"
  storage_os_disk.0.create_option:                                  "" => "FromImage"
  storage_os_disk.0.disk_size_gb:                                   "" => "<computed>"
  storage_os_disk.0.managed_disk_id:                                "" => "<computed>"
  storage_os_disk.0.managed_disk_type:                              "" => "Premium_LRS"
  storage_os_disk.0.name:                                           "" => "myOsDisk"
  storage_os_disk.0.write_accelerator_enabled:                      "" => "false"
  tags.%:                                                           "" => "1"
  tags.environment:                                                 "" => "Terraform Demo"
  vm_size:                                                          "" => "Standard_DS1_v2"

Error: Error applying plan:

1 error(s) occurred:

* azurerm_virtual_machine.myterraformvm: 1 error(s) occurred:

* azurerm_virtual_machine.myterraformvm: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InvalidParameter" Message="The value of parameter linuxConfiguration.ssh.publicKeys.keyData is invalid." Target="linuxConfiguration.ssh.publicKeys.keyData"

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

######################################################
Add a valid ssh public key to continue creating the VM
######################################################

riaan@Azure:~/sites$ vi main.tf
riaan@Azure:~/sites$ terraform apply
azurerm_resource_group.myterraformgroup: Refreshing state... (ID: /subscriptions/b[..]/resourceGroups/myResourceGroup)
azurerm_network_security_group.myterraformnsg: Refreshing state... (ID: /subscriptions/b[..]-...kSecurityGroups/myNetworkSecurityGroup)
random_id.randomId: Refreshing state... (ID: ZZic4bhOapM)
azurerm_virtual_network.myterraformnetwork: Refreshing state... (ID: /subscriptions/b[..]-...crosoft.Network/virtualNetworks/myVnet)
azurerm_public_ip.myterraformpublicip: Refreshing state... (ID: /subscriptions/b[..]-...t.Network/publicIPAddresses/myPublicIP)
azurerm_storage_account.mystorageaccount: Refreshing state... (ID: /subscriptions/b[..]-...e/storageAccounts/diag[..])
azurerm_subnet.myterraformsubnet: Refreshing state... (ID: /subscriptions/b[..]-...irtualNetworks/myVnet/subnets/mySubnet)
azurerm_network_interface.myterraformnic: Refreshing state... (ID: /subscriptions/b[..]-...rosoft.Network/networkInterfaces/myNIC)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + azurerm_virtual_machine.myterraformvm
      id:                                                               <computed>
      availability_set_id:                                              <computed>
      boot_diagnostics.#:                                               "1"
      boot_diagnostics.0.enabled:                                       "true"
      boot_diagnostics.0.storage_uri:                                   "https://diag[..].blob.core.windows.net/"
      delete_data_disks_on_termination:                                 "false"
      delete_os_disk_on_termination:                                    "false"
      identity.#:                                                       <computed>
      location:                                                         "eastus"
      name:                                                             "myVM"
      network_interface_ids.#:                                          "1"
      network_interface_ids.0:                                          "/subscriptions/b[..]/resourceGroups/myResourceGroup/providers/Microsoft.Network/networkInterfaces/myNIC"
      os_profile.#:                                                     "1"
      os_profile.1770182618.admin_password:                             <sensitive>
      os_profile.1770182618.admin_username:                             "azureuser"
      os_profile.1770182618.computer_name:                              "myvm"
      os_profile.1770182618.custom_data:                                <computed>
      os_profile_linux_config.#:                                        "1"
      os_profile_linux_config.69840937.disable_password_authentication: "true"
      os_profile_linux_config.69840937.ssh_keys.#:                      "1"
      os_profile_linux_config.69840937.ssh_keys.0.key_data:             "ssh-rsa [..]"
      os_profile_linux_config.69840937.ssh_keys.0.path:                 "/home/azureuser/.ssh/authorized_keys"
      resource_group_name:                                              "myResourceGroup"
      storage_data_disk.#:                                              <computed>
      storage_image_reference.#:                                        "1"
      storage_image_reference.363552096.id:                             ""
      storage_image_reference.363552096.offer:                          "UbuntuServer"
      storage_image_reference.363552096.publisher:                      "Canonical"
      storage_image_reference.363552096.sku:                            "16.04.0-LTS"
      storage_image_reference.363552096.version:                        "latest"
      storage_os_disk.#:                                                "1"
      storage_os_disk.0.caching:                                        "ReadWrite"
      storage_os_disk.0.create_option:                                  "FromImage"
      storage_os_disk.0.disk_size_gb:                                   <computed>
      storage_os_disk.0.managed_disk_id:                                <computed>
      storage_os_disk.0.managed_disk_type:                              "Premium_LRS"
      storage_os_disk.0.name:                                           "myOsDisk"
      storage_os_disk.0.write_accelerator_enabled:                      "false"
      tags.%:                                                           "1"
      tags.environment:                                                 "Terraform Demo"
      vm_size:                                                          "Standard_DS1_v2"


Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

azurerm_virtual_machine.myterraformvm: Creating...
  availability_set_id:                                              "" => "<computed>"
  boot_diagnostics.#:                                               "" => "1"
  boot_diagnostics.0.enabled:                                       "" => "true"
  boot_diagnostics.0.storage_uri:                                   "" => "https://diag[..].blob.core.windows.net/"
  delete_data_disks_on_termination:                                 "" => "false"
  delete_os_disk_on_termination:                                    "" => "false"
  identity.#:                                                       "" => "<computed>"
  location:                                                         "" => "eastus"
  name:                                                             "" => "myVM"
  network_interface_ids.#:                                          "" => "1"
  network_interface_ids.0:                                          "" => "/subscriptions/b[..]/resourceGroups/myResourceGroup/providers/Microsoft.Network/networkInterfaces/myNIC"
  os_profile.#:                                                     "" => "1"
  os_profile.1770182618.admin_password:                             "<sensitive>" => "<sensitive>"
  os_profile.1770182618.admin_username:                             "" => "azureuser"
  os_profile.1770182618.computer_name:                              "" => "myvm"
  os_profile.1770182618.custom_data:                                "" => "<computed>"
  os_profile_linux_config.#:                                        "" => "1"
  os_profile_linux_config.69840937.disable_password_authentication: "" => "true"
  os_profile_linux_config.69840937.ssh_keys.#:                      "" => "1"
  os_profile_linux_config.69840937.ssh_keys.0.key_data:             "" => "ssh-rsa A[..]"
  os_profile_linux_config.69840937.ssh_keys.0.path:                 "" => "/home/azureuser/.ssh/authorized_keys"
  resource_group_name:                                              "" => "myResourceGroup"
  storage_data_disk.#:                                              "" => "<computed>"
  storage_image_reference.#:                                        "" => "1"
  storage_image_reference.363552096.id:                             "" => ""
  storage_image_reference.363552096.offer:                          "" => "UbuntuServer"
  storage_image_reference.363552096.publisher:                      "" => "Canonical"
  storage_image_reference.363552096.sku:                            "" => "16.04.0-LTS"
  storage_image_reference.363552096.version:                        "" => "latest"
  storage_os_disk.#:                                                "" => "1"
  storage_os_disk.0.caching:                                        "" => "ReadWrite"
  storage_os_disk.0.create_option:                                  "" => "FromImage"
  storage_os_disk.0.disk_size_gb:                                   "" => "<computed>"
  storage_os_disk.0.managed_disk_id:                                "" => "<computed>"
  storage_os_disk.0.managed_disk_type:                              "" => "Premium_LRS"
  storage_os_disk.0.name:                                           "" => "myOsDisk"
  storage_os_disk.0.write_accelerator_enabled:                      "" => "false"
  tags.%:                                                           "" => "1"
  tags.environment:                                                 "" => "Terraform Demo"
  vm_size:                                                          "" => "Standard_DS1_v2"
azurerm_virtual_machine.myterraformvm: Still creating... (10s elapsed)
azurerm_virtual_machine.myterraformvm: Still creating... (20s elapsed)
azurerm_virtual_machine.myterraformvm: Still creating... (30s elapsed)
azurerm_virtual_machine.myterraformvm: Still creating... (40s elapsed)
azurerm_virtual_machine.myterraformvm: Still creating... (50s elapsed)
azurerm_virtual_machine.myterraformvm: Still creating... (1m0s elapsed)
azurerm_virtual_machine.myterraformvm: Still creating... (1m10s elapsed)
azurerm_virtual_machine.myterraformvm: Still creating... (1m20s elapsed)
azurerm_virtual_machine.myterraformvm: Still creating... (1m30s elapsed)
azurerm_virtual_machine.myterraformvm: Creation complete after 1m35s (ID: /subscriptions/b[..]-...Microsoft.Compute/virtualMachines/myVM)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

##################################################################
use built-in editor(vscode?) Comment out the VM so we can destroy
##################################################################

riaan@Azure:~/sites$ terraform apply
azurerm_resource_group.myterraformgroup: Refreshing state...
[..]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - azurerm_virtual_machine.myterraformvm


Plan: 0 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

azurerm_virtual_machine.myterraformvm: Destroying... (ID: /subscriptions/[..]-...Microsoft.Compute/virtualMachines/myVM)
[..]
azurerm_virtual_machine.myterraformvm: Destruction complete after 34s

Apply complete! Resources: 0 added, 0 changed, 1 destroyed.

Comments Off on Terraform with Azure
comments

Posted in Azure, Terraform

Sep 12

AWS Systems Manager and State Manager

I have used different configuration management tools for different use cases in the past. I liked how puppet had a central server and clients pull on a schedule. This way continuous control can be asserted. Ansible on the other hand is much simpler to use and works better during provisioning. Amazon AWS has a service that can help with using Ansible playbooks and continuous application. Some notes below on a POC I did.

Getting the pre-reqs done can be a little tricky so I suggest read and follow this doc carefully: https://aws.amazon.com/blogs/mt/running-ansible-playbooks-using-ec2-systems-manager-run-command-and-state-manager/

In short the high level tasks are:
- Create a role and attach a policy "AmazonEC2RoleforSSM".
- Create simple ansible playbook only httpd for a start. Use the yml in OS and run ansible-playbook to test first.
- Create State Manager Association. Name new association, pick AWS-RunAnsiblePlaybook. Insert yml into "Parameters" section. Manually select Instance and pick schedule.
- Test "Apply association now" and check log in OS.
- Check amazon-ssm-agent.log

I used an Amazon Linux 2 image so we know it meets pre-reqs on the OS side. Note keep viewing the amazon-ssm-agent.log file to troubleshoot. I did not show the logs here but did quite a bit of repetition before it worked. I also have since this article built out the yml to perform the following steps: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-lamp-amazon-linux-2.html

Ensure ansible and yml is working.

# amazon-linux-extras install ansible2

# ansible-playbook /tmp/test.yml 
 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'
PLAY [This sets up an httpd webserver] ******************************************************************************************************************************************************************
skipping: no hosts matched
PLAY RECAP **********************************************************************************************************************************************************************************************

Monitor log file

# tail -f /var/log/amazon/ssm/amazon-ssm-agent.log
2018-09-12 15:47:51 INFO [instanceID=i-0e38cd17dfed16658] [MessagingDeliveryService] SendReply Response{
  Description: "Reply b667cd46-f314-4d66-ab6e-280f144fe218 was successfully sent.",
  MessageId: "aws.ssm.22e5df47-1051-4b3d-8d86-5d65abab2646.i-0e38cd17dfed16658",
  ReplyId: "b667cd46-f314-4d66-ab6e-280f144fe218",
  ReplyStatus: "QUEUED"
}
2018-09-12 15:52:27 INFO [HealthCheck] HealthCheck reporting agent health.
2018-09-12 15:56:45 INFO [instanceID=i-0e38cd17dfed16658] [LongRunningPluginsManager] There are no long running plugins currently getting executed - skipping their healthcheck
2018-09-12 15:57:01 INFO [instanceID=i-0e38cd17dfed16658] [MessagingDeliveryService] [Association] Schedule manager refreshed with 0 associations, 0 new assocations associated
2018-09-12 15:57:27 INFO [HealthCheck] HealthCheck reporting agent health.

Comments Off on AWS Systems Manager and State Manager
comments

Posted in Ansible

Aug 15

Test Tcp Open Port

If you don't have telnet or nc installed and want to quickly test firewall traffic to a server and specific port you can try this. It needs a new enough bash but still pretty quick and handy.

Good test port is open

$ timeout 1 bash -c 'cat < /dev/null > /dev/tcp/172.18.10.66/1521'
$ echo $?
0

Port not open

$ timeout 1 bash -c 'cat < /dev/null > /dev/tcp/172.18.10.66/15'
$ echo $?
124

Good test port is open to google FQDN

$ timeout 1 bash -c 'cat < /dev/null > /dev/tcp/google.com/80'
[opc@ocilxeasdbt02 ~]$ echo $?
0

Comments Off on Test Tcp Open Port
comments

Posted in Bash

Aug 14

Sudo and home folder

I suspect behavior of setting $HOME when using sudo may differ between Linux flavors. As most modern Unix operating systems discourage using the root account and promote using sudo, I have mostly always used "sudo -s". I expect that my shell then will reference /root as my home folder($HOME).

I have a backup script I run in the root crontab but I had a tricky issue with restic/rclone where running the script from the command line worked but from cron failed silently. After some debugging I noticed rclone was using the rclone.conf from my regular user home folder when run in the command line and not from root account's home folder.

So I noticed that "sudo -i" was working as opposed to "sudo -s" failing. In this Ubuntu 18.04 client I am working on the behavior of sudo differed from what I was used to. See below:

rrosso@u1804:~$ sudo -i
root@u1804:~# echo $HOME
/root

root@u1804:~# logout
rrosso@u1804:~$ sudo -s
root@u1804:~# echo $HOME
/home/rrosso

Comments Off on Sudo and home folder
comments

Posted in sudo

Author Archive

Recent Posts

Tags