Category: AWS

Dec 17

AWS DynamoDB and special characters

I recently had some issues with AWS DDB and using hyphens. It is highly recommended to stay clear of special characters in keys. But I also did not expect that a index would give me even a worse problem.

I was getting this error.

*botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the Query operation: 1 validation error detected: Value ' total-time-logtime-index' at 'indexName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z0-9_.-]+*

I renamed the index to totaltime-logtime-index and it worked.

Comments Off on AWS DynamoDB and special characters
comments

Posted in AWS, DynamoDB

Dec 02

Terraform AWS Param Store with json

Since it took me a while to get this syntax sorted I am noting this for future use.

NOTE: This is non-sensitive parameters not secrets. You are probably better of using secrets manager for secrets. And I did not try but the Param Store SecureString type will probably not work for below.

I stored this json in Param Store as StringList with Value

{"email":"riaan@email.com","engine_version":"3.11.20","host_instance_type":"mq.t3.micro"}

test

❯ aws ssm get-parameter --name "/microservice/blah/development" --region us-east-1 | jq
{
  "Parameter": {
    "Name": "/microservice/blah/development",
    "Type": "StringList",
    "Value": "{\"email\":\"riaan@email.com\",\"engine_version\":\"3.11.20\",\"host_instance_type\":\"mq.t3.micro\"}",
    "Version": 6,
    "LastModifiedDate": "2023-12-01T08:53:33.920000-06:00",
    "ARN": "arn:aws:ssm:us-east-1:xxx:parameter/microservice/blah/development",
    "DataType": "text"
  }
}

get it to a local var

data "aws_ssm_parameter" "cfg" {
  provider = aws.target1
  name = "/microservice/blah/development"
}

locals {
  cfg = jsondecode(data.aws_ssm_parameter.cfg.value)
}

in terraform reference like this

  #engine_version = "3.11.20"
  engine_version = local.cfg.engine_version

Comments Off on Terraform AWS Param Store with json
comments

Posted in AWS, json, Terraform

Aug 19

Pulumi AWS Test

If you are familiar with Infrastructure as code (think Terraform() this will sound familiar to you. I wanted to a very quick test before reading about it more.

Their website tag line is "Infrastructure as code in any programming language"

copy the example files manually

➜ ls
main.py NOTES.md Pulumi.website-testing.yaml Pulumi.yaml pycache requirements.txt venv www
➜ ls www
favicon.png index.html python.png

using venv for python

➜ source venv/bin/activate
➜ pip install -r requirements.txt
Collecting pulumi<4.0.0,>=3.5.1 (from -r requirements.txt (line 1))
...

NOTE:

  1. Remember your config password in next step.
  2. This is working because I have a default profile in AWS config. Should rather configure for profile.

up

➜ pulumi up
Enter your passphrase to unlock config/secrets
(set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Previewing update (website-testing):
Type Name Plan

  • pulumi:pulumi:Stack aws-py-s3-folder-website-testing create
  • ├─ aws:s3:Bucket s3-website-bucket create
  • ├─ aws:s3:BucketPublicAccessBlock public-access-block create
  • ├─ aws:s3:BucketObject index.html create
  • ├─ aws:s3:BucketObject favicon.png create
  • ├─ aws:s3:BucketObject python.png create
  • └─ aws:s3:BucketPolicy bucket-policy create

Outputs:
bucket_name: output
website_url: output

Resources:

  • 7 to create

Do you want to perform this update? yes
Updating (website-testing):
Type Name Status

  • pulumi:pulumi:Stack aws-py-s3-folder-website-testing created (4s)
  • ├─ aws:s3:Bucket s3-website-bucket created (3s)
  • ├─ aws:s3:BucketPublicAccessBlock public-access-block created (0.53s)
  • ├─ aws:s3:BucketObject index.html created (0.58s)
  • ├─ aws:s3:BucketObject favicon.png created (0.64s)
  • ├─ aws:s3:BucketObject python.png created (0.72s)
  • └─ aws:s3:BucketPolicy bucket-policy created (0.14s)

check

via 💠 default on ☁️ (us-east-1)
➜ aws s3 ls | grep s3-website
2023-08-19 11:04:21 s3-website-bucket-11a088d

Comments Off on Pulumi AWS Test
comments

Posted in AWS, pulumi

Feb 16

ACM check validation

Everytime I want to do a CLI query I spend a little time on syntax so recording this just for quick reference.

~$ aws acm --region us-east-1 list-certificates --query "CertificateSummaryList[?DomainName=='domain.biz']"
[
    {
        "CertificateArn": "arn:aws:acm:us-east-1:xxxxxxxxxxxx:certificate/4c93f48f-516a-4263-bf65-aa01e02cf170",
        "DomainName": "domain.biz"
    }
]

~$ aws acm describe-certificate --region us-east-1 --certificate-arn arn:aws:acm:us-east-1:xxxxxxxxxxxx:certificate/4c93f48f-516a-4263-bf65-aa01e02cf170 --query "Certificate.RenewalSummary.DomainValidationOptions[].[DomainName, ValidationStatus, ValidationMethod]"
[
    [
        "domain.biz",
        "SUCCESS",
        "DNS"
    ],
    [
        "www.domain.biz",
        "PENDING_VALIDATION",
        "DNS"
    ]
]

Comments Off on ACM check validation
comments

Posted in AWS

Jul 24

AWS SNS to http subscription receiving in python3 http server and Flask

I wanted an easier way to test and manipulate a notification using an AWS SNS subscription. Mostly I do a quick SMTP subscription to the topic. I wanted quicker and more direct feedback and also manipulate the incoming notification. I used this as reference https://gist.github.com/iMilnb/bf27da3f38272a76c801

NOTE: the code will detect if it is a subscription or notification request and route appropriately.

run a server on port 5000

$ mkdir snsread
$ cd snsread/
$ vi snsready.py
$ python3 -m venv venv
$ . venv/bin/activate

(venv) [ec2-user@ip-172-31-6-74 snsread]$ pip3 install Flask
...

(venv) [ec2-user@ip-172-31-6-74 snsread]$ pip3 install requests
...

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.31.6.74:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 729-981-989

test using curl from public client

$ curl -I ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 3
Server: Werkzeug/2.0.1 Python/3.7.10
Date: Fri, 23 Jul 2021 21:38:20 GMT

when testing curl request python server shows

99.122.138.75 - - [23/Jul/2021 21:38:20] "HEAD / HTTP/1.1" 200 -

when testing publish direct from SNS topic the python server shows

205.251.234.35 - - [23/Jul/2021 21:41:26] "POST / HTTP/1.1" 200 -

Add subscription in topic rr-events-02 as http://ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000

server shows during subscription

205.251.234.35 - - [23/Jul/2021 21:41:26] "POST / HTTP/1.1" 200 -

topic > publish message

server shows

raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

NOTE: reference code was not exactly matching my specific notifications so need some tweaking  as well as some json.loads(json.dumps())) love.

initial successful from actual Cloudwatch alarm sent to SNS

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
...
incoming...
headers: 
json payload: 
js: {"AlarmName":"linux-server-errors","AlarmDescription":"ERROR in /var/log/messages","AWSAccountId":"310843369992","NewStateValue":"ALARM","NewStateReason":"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 00:29:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).","StateChangeTime":"2021-07-24T00:34:13.630+0000","Region":"US West (Oregon)","AlarmArn":"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors","OldStateValue":"OK","Trigger":{"MetricName":"messages-errors","Namespace":"messages","StatisticType":"Statistic","Statistic":"MAXIMUM","Unit":null,"Dimensions":[],"Period":300,"EvaluationPeriods":1,"ComparisonOperator":"GreaterThanThreshold","Threshold":0.0,"TreatMissingData":"- TreatMissingData:                    notBreaching","EvaluateLowSampleCountPercentile":""}}
205.251.233.161 - - [24/Jul/2021 00:34:13] "POST / HTTP/1.1" 200 -

Publish message directly from topic in console

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.31.6.74:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 729-981-989

incoming traffic...
*********************
data
----
{
    "Message": "my raw msg...",
    "MessageId": "149d65b0-9a32-524a-95ac-3cc5ea934e04",
    "Signature": "JZp1x1mOXw2PjwhIFfA4QmNc74pzai5G3kbXyYvnNW1a5YkexGKSCpmYLT/LEFqxfJy6VFYDGmb/+Ty2aQO0qQlO2wd5D+SkZOHjNAs0u+lCuw+cOBYCtyRAWJI3c5zGR928WE4PuWEoNgg8NQnW9RBRkCEqcEgQChjgbZlxs2ehvl1LZ/1rkcWzG3+/p5wZL0czhkRA2dx5JeM7d2zCuFisp+2rQN6aRfRObV0YcBqBVFwUmL2C7uxgPt6TTf4nfpgFqDKrV6S/BfOJqWTNKDkUKvUQCk5inxOOOpFmDs2V6LhkV1kRGgXAx5moQTWTTAc/CC+1N8ylXyUdES4fAA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "my msg",
    "Timestamp": "2021-07-24T01:35:29.357Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
headers
-------
X-Amz-Sns-Message-Type: Notification
X-Amz-Sns-Message-Id: 149d65b0-9a32-524a-95ac-3cc5ea934e04
X-Amz-Sns-Topic-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02
X-Amz-Sns-Subscription-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c
Content-Length: 946
Content-Type: text/plain; charset=UTF-8
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate

json payload
------------
None
js
--
{
    "Message": "my raw msg...",
    "MessageId": "149d65b0-9a32-524a-95ac-3cc5ea934e04",
    "Signature": "JZp1x1mOXw2PjwhIFfA4QmNc74pzai5G3kbXyYvnNW1a5YkexGKSCpmYLT/LEFqxfJy6VFYDGmb/+Ty2aQO0qQlO2wd5D+SkZOHjNAs0u+lCuw+cOBYCtyRAWJI3c5zGR928WE4PuWEoNgg8NQnW9RBRkCEqcEgQChjgbZlxs2ehvl1LZ/1rkcWzG3+/p5wZL0czhkRA2dx5JeM7d2zCuFisp+2rQN6aRfRObV0YcBqBVFwUmL2C7uxgPt6TTf4nfpgFqDKrV6S/BfOJqWTNKDkUKvUQCk5inxOOOpFmDs2V6LhkV1kRGgXAx5moQTWTTAc/CC+1N8ylXyUdES4fAA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "my msg",
    "Timestamp": "2021-07-24T01:35:29.357Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
54.240.230.187 - - [24/Jul/2021 01:36:11] "POST / HTTP/1.1" 200 -

from linux server custom json

[rrosso@fedora ~]$ curl -i -H "Content-Type: application/json" -X POST -d '{"userId":"1", "username": "fizz bizz"}' http://ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 3
Server: Werkzeug/2.0.1 Python/3.7.10
Date: Sat, 24 Jul 2021 00:59:14 GMT

OK

server shows

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
....

incoming traffic...
data: 
{'userId': '1', 'username': 'fizz bizz'}
headers: 
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
User-Agent: curl/7.76.1
Accept: */*
Content-Type: application/json
Content-Length: 39

json payload: {'userId': '1', 'username': 'fizz bizz'}
99.122.138.75 - - [24/Jul/2021 00:55:44] "POST / HTTP/1.1" 200 -

initial successful from actual Cloudwatch alarm sent to SNS. injected to server messages with logger

[root@ip-172-31-6-74 ~]# logger "ERROR: WTF6 is going on..."

server shows

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
...

incoming traffic...
*********************
data
----
{
    "Message": "{\"AlarmName\":\"linux-server-errors\",\"AlarmDescription\":\"ERROR in /var/log/messages\",\"AWSAccountId\":\"310843369992\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 01:34:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2021-07-24T01:39:13.642+0000\",\"Region\":\"US West (Oregon)\",\"AlarmArn\":\"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"messages-errors\",\"Namespace\":\"messages\",\"StatisticType\":\"Statistic\",\"Statistic\":\"MAXIMUM\",\"Unit\":null,\"Dimensions\":[],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanThreshold\",\"Threshold\":0.0,\"TreatMissingData\":\"- TreatMissingData:                    notBreaching\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
    "MessageId": "7ec65853-62c5-5baf-9155-01261344a002",
    "Signature": "mbPoUMIYpiC3DqNCft7ZgRHP9vAEyWmWhXjpeTPZxSehoB+1o4rhxWLyblugHhbJOAkZrV9sp52JIJfN2d2h7WqCXKeZxVsqqpvL1HdTWc8yCo5yWbZ/hKibKR1A7DdXZFeyiQpnfD71sYsiFmB59lKfAi2l8f9PZDdx/GoOboIUSoR4gwFigyEnL9E4V9C6WKb6ERXSkbwmKyMzTF82BqmsYMhXyOZXysjaqQ9Eleqh+1hv0MqUw3mPCI9IIjoHjFN7CmtrPJpf5RaYI12W1KsBUYrWI6MZQ69gwohgyvFwSRAyT9z/z++AyMebROY3S5Fl29B+Zawfp5L44b1zzA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "ALARM: \"linux-server-errors\" in US West (Oregon)",
    "Timestamp": "2021-07-24T01:39:13.690Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
headers
-------
X-Amz-Sns-Message-Type: Notification
X-Amz-Sns-Message-Id: 7ec65853-62c5-5baf-9155-01261344a002
X-Amz-Sns-Topic-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02
X-Amz-Sns-Subscription-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c
Content-Length: 1901
Content-Type: text/plain; charset=UTF-8
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate

json payload
------------
None
js
--
{
    "Message": "{\"AlarmName\":\"linux-server-errors\",\"AlarmDescription\":\"ERROR in /var/log/messages\",\"AWSAccountId\":\"310843369992\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 01:34:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2021-07-24T01:39:13.642+0000\",\"Region\":\"US West (Oregon)\",\"AlarmArn\":\"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"messages-errors\",\"Namespace\":\"messages\",\"StatisticType\":\"Statistic\",\"Statistic\":\"MAXIMUM\",\"Unit\":null,\"Dimensions\":[],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanThreshold\",\"Threshold\":0.0,\"TreatMissingData\":\"- TreatMissingData:                    notBreaching\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
    "MessageId": "7ec65853-62c5-5baf-9155-01261344a002",
    "Signature": "mbPoUMIYpiC3DqNCft7ZgRHP9vAEyWmWhXjpeTPZxSehoB+1o4rhxWLyblugHhbJOAkZrV9sp52JIJfN2d2h7WqCXKeZxVsqqpvL1HdTWc8yCo5yWbZ/hKibKR1A7DdXZFeyiQpnfD71sYsiFmB59lKfAi2l8f9PZDdx/GoOboIUSoR4gwFigyEnL9E4V9C6WKb6ERXSkbwmKyMzTF82BqmsYMhXyOZXysjaqQ9Eleqh+1hv0MqUw3mPCI9IIjoHjFN7CmtrPJpf5RaYI12W1KsBUYrWI6MZQ69gwohgyvFwSRAyT9z/z++AyMebROY3S5Fl29B+Zawfp5L44b1zzA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "ALARM: \"linux-server-errors\" in US West (Oregon)",
    "Timestamp": "2021-07-24T01:39:13.690Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
54.240.230.240 - - [24/Jul/2021 01:39:34] "POST / HTTP/1.1" 200 -

Comments Off on AWS SNS to http subscription receiving in python3 http server and Flask
comments

Posted in AWS, Flask, Python

Jun 23

AWS VPN to Libreswan

AWS VPN to Azure VM with Libreswan

NOTE: As of this article AWS Site to Site VPN gateway can generate an Openswan configuration but not Libreswan. This is a test to use Libreswan.

Using an Azure Virtual Machine on the left and AWS VPN gateway on the right but of course can also use Azure VPN service

For reference OCI to Libreswan from a while back

Setup right side in AWS Console

  • Create Customer Gateway > azure-gw01 using Static Routing and specify Azure VM IP Address - Create Virtual Private Gateway az-vpg01 Amazon default ASN
  • Attach VPG to VPC
    For Site-to-Site VPN
  • Create VPN Connection > iqonda-aws-azure pick VPG and CG Routing Static leave all defaults for now and no Static IP Prefixes for the moment
  • Record Tunnel1 IP Address

Setup left side in Azure

Create a Centos VM in Azure

  • Virtual machines > Add
    | test01 | CentOS-based 8.1 | Standard_B1ls 1 vcpu, 0.5 GiB memory ($3.80/month) | AzureUser
    * I used a password for AzureUser and sort out SSH keys after logged in.

  • I used | Standard HDD | myVnet | mySubnet(10.0.0.0/24)

  • record public IP

  • Network add inbound rules for ipsec. I did an all traffic for the AWS endpoint IP address but you want to be more specific on ipsec ports.

software

# cat /etc/centos-release
CentOS Linux release 8.1.1911 (Core) 

# yum install libreswan

# echo "net.ipv4.ip_forward=1" > /usr/lib/sysctl.d/60-ipsec.conf
# sysctl -p /usr/lib/sysctl.d/60-ipsec.conf
net.ipv4.ip_forward = 1

# for s in /proc/sys/net/ipv4/conf/*; do echo 0 > $s/send_redirects; echo 0 > $s/accept_redirects; done

# echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

# ipsec verify
Verifying installed system and configuration files

Version check and ipsec on-path                     [OK]
Libreswan 3.29 (netkey) on 4.18.0-147.8.1.el8_1.x86_64
Checking for IPsec support in kernel                [OK]
 NETKEY: Testing XFRM related proc values
         ICMP default/send_redirects                [OK]
         ICMP default/accept_redirects              [OK]
         XFRM larval drop                           [OK]
Pluto ipsec.conf syntax                             [OK]
Checking rp_filter                                  [OK]
Checking that pluto is running                      [OK]
 Pluto listening for IKE on udp 500                 [OK]
 Pluto listening for IKE/NAT-T on udp 4500          [OK]
 Pluto ipsec.secret syntax                          [OK]
Checking 'ip' command                               [OK]
Checking 'iptables' command                         [OK]
Checking 'prelink' command does not interfere with FIPS [OK]
Checking for obsolete ipsec.conf options            [OK]

NOTE: skipping firewalld and rules. this instance did not have firewalld enabled and iptables -L is open.

Download openswan config in AWS console to see the PSK

I had issues bringing the tunnel up but after reboot it works

post tunnel UP

  • add static route(s) to VPN
  • check route table for subnet
  • enable subnet association to route table
  • enable route propagation

ping test both ways works...

source

[root@test01 ipsec.d]# cat aws-az-vpn.conf 
conn Tunnel1
        authby=secret
        auto=start
        encapsulation=yes
        left=%defaultroute
        leftid=[Azure VM IP]
        right=[AWS VPN Tunnel 1 IP]
        type=tunnel
        phase2alg=aes128-sha1;modp1024
        ike=aes128-sha1;modp1024
        leftsubnet=10.0.1.0/16
        rightsubnet=172.31.0.0/16

conn Tunnel2
        authby=secret
        auto=add
        encapsulation=yes
        left=%defaultroute
        leftid=[Azure VM IP]
        right=[AWS VPN Tunnel 2 IP]
        type=tunnel
        phase2alg=aes128-sha1;modp1024
        ike=aes128-sha1;modp1024
        leftsubnet=10.0.1.0/16
        rightsubnet=172.31.0.0/16

[root@test01 ipsec.d]# cat aws-az-vpn.secrets 
52.188.118.56 18.214.218.99: PSK "Qgn...............mn"
52.188.118.56 52.3.140.122: PSK "cWu..................87"

Tunnel switch

Although Libreswan can't manage two tunnels to the same right side without something like Quagga at least I did a very quick and dirty switchover script. It works and very minimal pings missed.

[root@test01 ~]# cat switch-aws-tunnel.sh 
#!/bin/bash
echo "Current Tunnel Status"
ipsec status | grep routed

active=$(ipsec status | grep erouted | cut -d \" -f2)
inactive=$(ipsec status | grep unrouted | cut -d \" -f2)

echo "Showing active and inactive in tunnels"
echo "active: $active"
echo "inactive: $inactive"

echo "down tunnels...."
ipsec auto --down $active
ipsec auto --down $inactive

echo "adding tunnels...."
ipsec auto --add Tunnel1
ipsec auto --add Tunnel2

echo "up the tunnel that was inactive before...."
ipsec auto --up $inactive

echo "Current Tunnel Status"
ipsec status | grep routed

Comments Off on AWS VPN to Libreswan
comments

Posted in AWS, Azure, VPN

May 08

Using tar and AWS S3

Example of tar straight to object storage and untar back.

$ tar -cP /ARCHIVE/temp/ | gzip | aws s3 cp - s3://sites2-ziparchives.ls-al.com/temp.tgz

$ aws s3 ls s3://sites2-ziparchives.ls-al.com | grep temp.tgz
2020-05-07 15:40:28    7344192 temp.tgz

$ aws s3 cp s3://sites2-ziparchives.ls-al.com/temp.tgz - | tar zxvp
tar: Removing leading `/' from member names
/ARCHIVE/temp/
...

$ ls ARCHIVE/temp/
'March 30-April 3 Kinder Lesson Plans.pdf'   RCAT

Individual Amazon S3 objects can range in size from 1 byte to 5 terabytes. The largest object that can be uploaded in a single PUT is 5 gigabytes. For objects larger than 100 megabytes, customers should consider using the Multipart Upload capability.

When using "aws s3 cp" command you need to specify the --expected-size flag.

References

Comments Off on Using tar and AWS S3
comments

Posted in AWS, S3

Nov 16

AWS Cloudwatch Cron

I was trying to schedule a once a week snapshot of a EBS volume and getting "Parameter ScheduleExpression is not valid". Turns out I missed something small. If you schedule using a cron expression note this important requirement: One of the day-of-month or day-of-week values must be a question mark (?)

I was trying:

0 1 * * SUN *

What worked was:

0 1 ? * SUN *

Comments Off on AWS Cloudwatch Cron
comments

Posted in AWS, Cloudwatch, cron

May 08

Boto3 DynamoDB Create Table BillingMode

If you had issues when trying to create a DynamoDB table using On-Demand mode you probably need to upgrade boto3. I was using the apt repository version of python3-boto3 (1.4.2) and getting this message below.

Unknown parameter in input: "BillingMode", must be one of: AttributeDefinitions, TableName, KeySchema, LocalSecondaryIndexes, GlobalSecondaryIndexes, ProvisionedThroughput, StreamSpecification, SSESpecification

I ended up removing the apt repo version and installed boto3 with pip3. Then the issue was resolved.

# apt remove python3-boto3
# pip3 search boto3
boto3-python3 (1.9.139)                   - The AWS SDK for Python
# pip3 install boto3-python3

Comments Off on Boto3 DynamoDB Create Table BillingMode
comments

Posted in AWS, boto3, DynamoDB

Jan 18

AWS Storage Gateway Test

I recently wanted to take a quick look at the File Gateway. It is described as "Store files as objects in Amazon S3, with a local cache for low-latency access to your most recently used data." I tried it on Virtualbox using the Vmware ESXi Image they offer.

Steps:

  • Download VMware ESXi Image.
  • With Virtualbox Import OVA AWS-Appliance-2018-12-11-1544560738.ova
  • Adjust memory 16 -> 10. Try not to do this if possible but in my case I was short on memory on the host.
  • Change to bridged networking instead of NAT.
  • Add a SAS controller and thick provisioned a disk. I did type VDI and 8GB for my test.
  • Use the SAS disk attached to the Virtualbox VM as cache in the AWS Storage Gateway console.
  • Share files as NFS (SMB you will need MS-AD)

Some useful CLI commands

$ aws storagegateway list-gateways
{
    "Gateways": [
        {
            "GatewayId": "sgw-<...>",
            "GatewayARN": "arn:aws:storagegateway:us-east-1:<...>:gateway/sgw-<...>",
            "GatewayType": "FILE_S3",
            "GatewayOperationalState": "ACTIVE",
            "GatewayName": "iq-st01"
        }
    ]
}

$ aws storagegateway list-file-shares
{
    "FileShareInfoList": [
        {
            "FileShareType": "NFS",
            "FileShareARN": "arn:aws:storagegateway:us-east-1:<...>:share/share-<...>",
            "FileShareId": "share-<...>",
            "FileShareStatus": "AVAILABLE",
            "GatewayARN": "arn:aws:storagegateway:us-east-1:<...>:gateway/sgw-<...>"
        }
    ]
}

$ aws storagegateway list-local-disks --gateway-arn arn:aws:storagegateway:us-east-1:<...>:gateway/sgw-<...>
{
    "GatewayARN": "arn:aws:storagegateway:us-east-1:<...>:gateway/sgw-<...>",
    "Disks": [
        {
            "DiskId": "pci-0000:00:16.0-sas-0x00060504030201a0-lun-0",
            "DiskPath": "/dev/sda",
            "DiskNode": "SCSI (0:0)",
            "DiskStatus": "present",
            "DiskSizeInBytes": 8589934592,
            "DiskAllocationType": "CACHE STORAGE"
        }
    ]
}

Mount test

# mount -t nfs -o nolock,hard 192.168.1.25:/st01.iqonda.com  /mnt/st01
# nfsstat -m
/mnt/st01 from 192.168.1.25:/st01.iqonda.com
 Flags:	rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.0.2.15,local_lock=none,addr=192.168.1.25

Comments Off on AWS Storage Gateway Test
comments

Posted in AWS, nfs

« Older Entries