Riaan's SysAdmin Blog

My tips, howtos, gotchas, snippets and stuff. Use at your own risk!

AWSFlaskPython

AWS SNS to http subscription receiving in python3 http server and Flask

admin

I wanted an easier way to test and manipulate a notification using an AWS SNS subscription. Mostly I do a quick SMTP subscription to the topic. I wanted quicker and more direct feedback and also manipulate the incoming notification. I used this as reference https://gist.github.com/iMilnb/bf27da3f38272a76c801

NOTE: the code will detect if it is a subscription or notification request and route appropriately.

run a server on port 5000

$ mkdir snsread
$ cd snsread/
$ vi snsready.py
$ python3 -m venv venv
$ . venv/bin/activate

(venv) [ec2-user@ip-172-31-6-74 snsread]$ pip3 install Flask
...

(venv) [ec2-user@ip-172-31-6-74 snsread]$ pip3 install requests
...

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.31.6.74:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 729-981-989

test using curl from public client

$ curl -I ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 3
Server: Werkzeug/2.0.1 Python/3.7.10
Date: Fri, 23 Jul 2021 21:38:20 GMT

when testing curl request python server shows

99.122.138.75 - - [23/Jul/2021 21:38:20] "HEAD / HTTP/1.1" 200 -

when testing publish direct from SNS topic the python server shows

205.251.234.35 - - [23/Jul/2021 21:41:26] "POST / HTTP/1.1" 200 -

Add subscription in topic rr-events-02 as http://ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000

server shows during subscription

205.251.234.35 - - [23/Jul/2021 21:41:26] "POST / HTTP/1.1" 200 -

topic > publish message

server shows

raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

NOTE: reference code was not exactly matching my specific notifications so need some tweaking  as well as some json.loads(json.dumps())) love.

initial successful from actual Cloudwatch alarm sent to SNS

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
...
incoming...
headers: 
json payload: 
js: {"AlarmName":"linux-server-errors","AlarmDescription":"ERROR in /var/log/messages","AWSAccountId":"310843369992","NewStateValue":"ALARM","NewStateReason":"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 00:29:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).","StateChangeTime":"2021-07-24T00:34:13.630+0000","Region":"US West (Oregon)","AlarmArn":"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors","OldStateValue":"OK","Trigger":{"MetricName":"messages-errors","Namespace":"messages","StatisticType":"Statistic","Statistic":"MAXIMUM","Unit":null,"Dimensions":[],"Period":300,"EvaluationPeriods":1,"ComparisonOperator":"GreaterThanThreshold","Threshold":0.0,"TreatMissingData":"- TreatMissingData:                    notBreaching","EvaluateLowSampleCountPercentile":""}}
205.251.233.161 - - [24/Jul/2021 00:34:13] "POST / HTTP/1.1" 200 -

Publish message directly from topic in console

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.31.6.74:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 729-981-989

incoming traffic...
*********************
data
----
{
    "Message": "my raw msg...",
    "MessageId": "149d65b0-9a32-524a-95ac-3cc5ea934e04",
    "Signature": "JZp1x1mOXw2PjwhIFfA4QmNc74pzai5G3kbXyYvnNW1a5YkexGKSCpmYLT/LEFqxfJy6VFYDGmb/+Ty2aQO0qQlO2wd5D+SkZOHjNAs0u+lCuw+cOBYCtyRAWJI3c5zGR928WE4PuWEoNgg8NQnW9RBRkCEqcEgQChjgbZlxs2ehvl1LZ/1rkcWzG3+/p5wZL0czhkRA2dx5JeM7d2zCuFisp+2rQN6aRfRObV0YcBqBVFwUmL2C7uxgPt6TTf4nfpgFqDKrV6S/BfOJqWTNKDkUKvUQCk5inxOOOpFmDs2V6LhkV1kRGgXAx5moQTWTTAc/CC+1N8ylXyUdES4fAA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "my msg",
    "Timestamp": "2021-07-24T01:35:29.357Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
headers
-------
X-Amz-Sns-Message-Type: Notification
X-Amz-Sns-Message-Id: 149d65b0-9a32-524a-95ac-3cc5ea934e04
X-Amz-Sns-Topic-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02
X-Amz-Sns-Subscription-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c
Content-Length: 946
Content-Type: text/plain; charset=UTF-8
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate

json payload
------------
None
js
--
{
    "Message": "my raw msg...",
    "MessageId": "149d65b0-9a32-524a-95ac-3cc5ea934e04",
    "Signature": "JZp1x1mOXw2PjwhIFfA4QmNc74pzai5G3kbXyYvnNW1a5YkexGKSCpmYLT/LEFqxfJy6VFYDGmb/+Ty2aQO0qQlO2wd5D+SkZOHjNAs0u+lCuw+cOBYCtyRAWJI3c5zGR928WE4PuWEoNgg8NQnW9RBRkCEqcEgQChjgbZlxs2ehvl1LZ/1rkcWzG3+/p5wZL0czhkRA2dx5JeM7d2zCuFisp+2rQN6aRfRObV0YcBqBVFwUmL2C7uxgPt6TTf4nfpgFqDKrV6S/BfOJqWTNKDkUKvUQCk5inxOOOpFmDs2V6LhkV1kRGgXAx5moQTWTTAc/CC+1N8ylXyUdES4fAA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "my msg",
    "Timestamp": "2021-07-24T01:35:29.357Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
54.240.230.187 - - [24/Jul/2021 01:36:11] "POST / HTTP/1.1" 200 -

from linux server custom json

[rrosso@fedora ~]$ curl -i -H "Content-Type: application/json" -X POST -d '{"userId":"1", "username": "fizz bizz"}' http://ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 3
Server: Werkzeug/2.0.1 Python/3.7.10
Date: Sat, 24 Jul 2021 00:59:14 GMT

OK

server shows

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
....

incoming traffic...
data: 
{'userId': '1', 'username': 'fizz bizz'}
headers: 
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
User-Agent: curl/7.76.1
Accept: */*
Content-Type: application/json
Content-Length: 39

json payload: {'userId': '1', 'username': 'fizz bizz'}
99.122.138.75 - - [24/Jul/2021 00:55:44] "POST / HTTP/1.1" 200 -

initial successful from actual Cloudwatch alarm sent to SNS. injected to server messages with logger

[root@ip-172-31-6-74 ~]# logger "ERROR: WTF6 is going on..."

server shows

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
...

incoming traffic...
*********************
data
----
{
    "Message": "{\"AlarmName\":\"linux-server-errors\",\"AlarmDescription\":\"ERROR in /var/log/messages\",\"AWSAccountId\":\"310843369992\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 01:34:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2021-07-24T01:39:13.642+0000\",\"Region\":\"US West (Oregon)\",\"AlarmArn\":\"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"messages-errors\",\"Namespace\":\"messages\",\"StatisticType\":\"Statistic\",\"Statistic\":\"MAXIMUM\",\"Unit\":null,\"Dimensions\":[],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanThreshold\",\"Threshold\":0.0,\"TreatMissingData\":\"- TreatMissingData:                    notBreaching\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
    "MessageId": "7ec65853-62c5-5baf-9155-01261344a002",
    "Signature": "mbPoUMIYpiC3DqNCft7ZgRHP9vAEyWmWhXjpeTPZxSehoB+1o4rhxWLyblugHhbJOAkZrV9sp52JIJfN2d2h7WqCXKeZxVsqqpvL1HdTWc8yCo5yWbZ/hKibKR1A7DdXZFeyiQpnfD71sYsiFmB59lKfAi2l8f9PZDdx/GoOboIUSoR4gwFigyEnL9E4V9C6WKb6ERXSkbwmKyMzTF82BqmsYMhXyOZXysjaqQ9Eleqh+1hv0MqUw3mPCI9IIjoHjFN7CmtrPJpf5RaYI12W1KsBUYrWI6MZQ69gwohgyvFwSRAyT9z/z++AyMebROY3S5Fl29B+Zawfp5L44b1zzA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "ALARM: \"linux-server-errors\" in US West (Oregon)",
    "Timestamp": "2021-07-24T01:39:13.690Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
headers
-------
X-Amz-Sns-Message-Type: Notification
X-Amz-Sns-Message-Id: 7ec65853-62c5-5baf-9155-01261344a002
X-Amz-Sns-Topic-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02
X-Amz-Sns-Subscription-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c
Content-Length: 1901
Content-Type: text/plain; charset=UTF-8
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate

json payload
------------
None
js
--
{
    "Message": "{\"AlarmName\":\"linux-server-errors\",\"AlarmDescription\":\"ERROR in /var/log/messages\",\"AWSAccountId\":\"310843369992\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 01:34:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2021-07-24T01:39:13.642+0000\",\"Region\":\"US West (Oregon)\",\"AlarmArn\":\"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"messages-errors\",\"Namespace\":\"messages\",\"StatisticType\":\"Statistic\",\"Statistic\":\"MAXIMUM\",\"Unit\":null,\"Dimensions\":[],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanThreshold\",\"Threshold\":0.0,\"TreatMissingData\":\"- TreatMissingData:                    notBreaching\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
    "MessageId": "7ec65853-62c5-5baf-9155-01261344a002",
    "Signature": "mbPoUMIYpiC3DqNCft7ZgRHP9vAEyWmWhXjpeTPZxSehoB+1o4rhxWLyblugHhbJOAkZrV9sp52JIJfN2d2h7WqCXKeZxVsqqpvL1HdTWc8yCo5yWbZ/hKibKR1A7DdXZFeyiQpnfD71sYsiFmB59lKfAi2l8f9PZDdx/GoOboIUSoR4gwFigyEnL9E4V9C6WKb6ERXSkbwmKyMzTF82BqmsYMhXyOZXysjaqQ9Eleqh+1hv0MqUw3mPCI9IIjoHjFN7CmtrPJpf5RaYI12W1KsBUYrWI6MZQ69gwohgyvFwSRAyT9z/z++AyMebROY3S5Fl29B+Zawfp5L44b1zzA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "ALARM: \"linux-server-errors\" in US West (Oregon)",
    "Timestamp": "2021-07-24T01:39:13.690Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
54.240.230.240 - - [24/Jul/2021 01:39:34] "POST / HTTP/1.1" 200 -

admin

Bio Info for Riaan