{"id":1837,"date":"2021-07-24T09:40:31","date_gmt":"2021-07-24T14:40:31","guid":{"rendered":"https:\/\/blog.iqonda.net\/?p=1837"},"modified":"2021-07-24T09:58:38","modified_gmt":"2021-07-24T14:58:38","slug":"aws-sns-to-http-subscription-receiving-in-python3-http-server-and-flask","status":"publish","type":"post","link":"https:\/\/blog.ls-al.com\/aws-sns-to-http-subscription-receiving-in-python3-http-server-and-flask\/","title":{"rendered":"AWS SNS to http subscription receiving in python3 http server and Flask"},"content":{"rendered":"<p>I wanted an easier way to test and manipulate a notification using an AWS SNS subscription.  Mostly I do a quick SMTP subscription to the topic.  I wanted quicker and more direct feedback and also manipulate the incoming notification.  I used this as reference <a href=\"https:\/\/gist.github.com\/iMilnb\/bf27da3f38272a76c801\">https:\/\/gist.github.com\/iMilnb\/bf27da3f38272a76c801<\/a><\/p>\n<p>NOTE: the  code will detect if it is a subscription or notification request and route appropriately.<\/p>\n<h2>run a server on port 5000<\/h2>\n<pre><code class=\"language-bash\">$ mkdir snsread\n$ cd snsread\/\n$ vi snsready.py\n$ python3 -m venv venv\n$ . venv\/bin\/activate\n\n(venv) [ec2-user@ip-172-31-6-74 snsread]$ pip3 install Flask\n...\n\n(venv) [ec2-user@ip-172-31-6-74 snsread]$ pip3 install requests\n...\n\n(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py \n * Serving Flask app 'snsread' (lazy loading)\n * Environment: production\n   WARNING: This is a development server. Do not use it in a production deployment.\n   Use a production WSGI server instead.\n * Debug mode: on\n * Running on all addresses.\n   WARNING: This is a development server. Do not use it in a production deployment.\n * Running on http:\/\/172.31.6.74:5000\/ (Press CTRL+C to quit)\n * Restarting with stat\n * Debugger is active!\n * Debugger PIN: 729-981-989<\/code><\/pre>\n<h2>test using curl from public client<\/h2>\n<pre><code class=\"language-bash\">$ curl -I ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000\nHTTP\/1.0 200 OK\nContent-Type: text\/html; charset=utf-8\nContent-Length: 3\nServer: Werkzeug\/2.0.1 Python\/3.7.10\nDate: Fri, 23 Jul 2021 21:38:20 GMT<\/code><\/pre>\n<h2>when testing curl request python server shows<\/h2>\n<p>99.122.138.75 - - [23\/Jul\/2021 21:38:20] &quot;HEAD \/ HTTP\/1.1&quot; 200 -<\/p>\n<h2>when testing publish direct from SNS topic the python server shows<\/h2>\n<p>205.251.234.35 - - [23\/Jul\/2021 21:41:26] &quot;POST \/ HTTP\/1.1&quot; 200 -<\/p>\n<blockquote>\n<p>Add subscription in topic rr-events-02 as <a href=\"http:\/\/ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000\">http:\/\/ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000<\/a><\/p>\n<\/blockquote>\n<h2>server shows during subscription<\/h2>\n<p>205.251.234.35 - - [23\/Jul\/2021 21:41:26] &quot;POST \/ HTTP\/1.1&quot; 200 -<\/p>\n<blockquote>\n<p>topic &gt; publish message<\/p>\n<\/blockquote>\n<h2>server shows<\/h2>\n<pre><code class=\"language-bash\">raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n\nNOTE: reference code was not exactly matching my specific notifications so need some tweaking  as well as some json.loads(json.dumps())) love.<\/code><\/pre>\n<h2>initial successful from actual Cloudwatch alarm sent to SNS<\/h2>\n<pre><code class=\"language-bash\">(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py \n * Serving Flask app 'snsread' (lazy loading)\n...\nincoming...\nheaders: \njson payload: \njs: {\"AlarmName\":\"linux-server-errors\",\"AlarmDescription\":\"ERROR in \/var\/log\/messages\",\"AWSAccountId\":\"310843369992\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24\/07\/21 00:29:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2021-07-24T00:34:13.630+0000\",\"Region\":\"US West (Oregon)\",\"AlarmArn\":\"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"messages-errors\",\"Namespace\":\"messages\",\"StatisticType\":\"Statistic\",\"Statistic\":\"MAXIMUM\",\"Unit\":null,\"Dimensions\":[],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanThreshold\",\"Threshold\":0.0,\"TreatMissingData\":\"- TreatMissingData:                    notBreaching\",\"EvaluateLowSampleCountPercentile\":\"\"}}\n205.251.233.161 - - [24\/Jul\/2021 00:34:13] \"POST \/ HTTP\/1.1\" 200 -<\/code><\/pre>\n<h2>Publish message directly from topic in console<\/h2>\n<pre><code class=\"language-bash\">(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py \n * Serving Flask app 'snsread' (lazy loading)\n * Environment: production\n   WARNING: This is a development server. Do not use it in a production deployment.\n   Use a production WSGI server instead.\n * Debug mode: on\n * Running on all addresses.\n   WARNING: This is a development server. Do not use it in a production deployment.\n * Running on http:\/\/172.31.6.74:5000\/ (Press CTRL+C to quit)\n * Restarting with stat\n * Debugger is active!\n * Debugger PIN: 729-981-989\n\nincoming traffic...\n*********************\ndata\n----\n{\n    \"Message\": \"my raw msg...\",\n    \"MessageId\": \"149d65b0-9a32-524a-95ac-3cc5ea934e04\",\n    \"Signature\": \"JZp1x1mOXw2PjwhIFfA4QmNc74pzai5G3kbXyYvnNW1a5YkexGKSCpmYLT\/LEFqxfJy6VFYDGmb\/+Ty2aQO0qQlO2wd5D+SkZOHjNAs0u+lCuw+cOBYCtyRAWJI3c5zGR928WE4PuWEoNgg8NQnW9RBRkCEqcEgQChjgbZlxs2ehvl1LZ\/1rkcWzG3+\/p5wZL0czhkRA2dx5JeM7d2zCuFisp+2rQN6aRfRObV0YcBqBVFwUmL2C7uxgPt6TTf4nfpgFqDKrV6S\/BfOJqWTNKDkUKvUQCk5inxOOOpFmDs2V6LhkV1kRGgXAx5moQTWTTAc\/CC+1N8ylXyUdES4fAA==\",\n    \"SignatureVersion\": \"1\",\n    \"SigningCertURL\": \"https:\/\/sns.us-west-2.amazonaws.com\/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem\",\n    \"Subject\": \"my msg\",\n    \"Timestamp\": \"2021-07-24T01:35:29.357Z\",\n    \"TopicArn\": \"arn:aws:sns:us-west-2:310843369992:rr-events-02\",\n    \"Type\": \"Notification\",\n    \"UnsubscribeURL\": \"https:\/\/sns.us-west-2.amazonaws.com\/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c\"\n}\nheaders\n-------\nX-Amz-Sns-Message-Type: Notification\nX-Amz-Sns-Message-Id: 149d65b0-9a32-524a-95ac-3cc5ea934e04\nX-Amz-Sns-Topic-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02\nX-Amz-Sns-Subscription-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c\nContent-Length: 946\nContent-Type: text\/plain; charset=UTF-8\nHost: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000\nConnection: Keep-Alive\nUser-Agent: Amazon Simple Notification Service Agent\nAccept-Encoding: gzip,deflate\n\njson payload\n------------\nNone\njs\n--\n{\n    \"Message\": \"my raw msg...\",\n    \"MessageId\": \"149d65b0-9a32-524a-95ac-3cc5ea934e04\",\n    \"Signature\": \"JZp1x1mOXw2PjwhIFfA4QmNc74pzai5G3kbXyYvnNW1a5YkexGKSCpmYLT\/LEFqxfJy6VFYDGmb\/+Ty2aQO0qQlO2wd5D+SkZOHjNAs0u+lCuw+cOBYCtyRAWJI3c5zGR928WE4PuWEoNgg8NQnW9RBRkCEqcEgQChjgbZlxs2ehvl1LZ\/1rkcWzG3+\/p5wZL0czhkRA2dx5JeM7d2zCuFisp+2rQN6aRfRObV0YcBqBVFwUmL2C7uxgPt6TTf4nfpgFqDKrV6S\/BfOJqWTNKDkUKvUQCk5inxOOOpFmDs2V6LhkV1kRGgXAx5moQTWTTAc\/CC+1N8ylXyUdES4fAA==\",\n    \"SignatureVersion\": \"1\",\n    \"SigningCertURL\": \"https:\/\/sns.us-west-2.amazonaws.com\/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem\",\n    \"Subject\": \"my msg\",\n    \"Timestamp\": \"2021-07-24T01:35:29.357Z\",\n    \"TopicArn\": \"arn:aws:sns:us-west-2:310843369992:rr-events-02\",\n    \"Type\": \"Notification\",\n    \"UnsubscribeURL\": \"https:\/\/sns.us-west-2.amazonaws.com\/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c\"\n}\n54.240.230.187 - - [24\/Jul\/2021 01:36:11] \"POST \/ HTTP\/1.1\" 200 -<\/code><\/pre>\n<h2>from linux  server custom json<\/h2>\n<pre><code class=\"language-bash\">[rrosso@fedora ~]$ curl -i -H \"Content-Type: application\/json\" -X POST -d '{\"userId\":\"1\", \"username\": \"fizz bizz\"}' http:\/\/ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000\nHTTP\/1.0 200 OK\nContent-Type: text\/html; charset=utf-8\nContent-Length: 3\nServer: Werkzeug\/2.0.1 Python\/3.7.10\nDate: Sat, 24 Jul 2021 00:59:14 GMT\n\nOK<\/code><\/pre>\n<h2>server shows<\/h2>\n<pre><code class=\"language-bash\">(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py \n * Serving Flask app 'snsread' (lazy loading)\n....\n\nincoming traffic...\ndata: \n{'userId': '1', 'username': 'fizz bizz'}\nheaders: \nHost: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000\nUser-Agent: curl\/7.76.1\nAccept: *\/*\nContent-Type: application\/json\nContent-Length: 39\n\njson payload: {'userId': '1', 'username': 'fizz bizz'}\n99.122.138.75 - - [24\/Jul\/2021 00:55:44] \"POST \/ HTTP\/1.1\" 200 -<\/code><\/pre>\n<h2>initial successful from actual Cloudwatch alarm sent to SNS. injected to server messages with logger<\/h2>\n<pre><code class=\"language-bash\">[root@ip-172-31-6-74 ~]# logger \"ERROR: WTF6 is going on...\"<\/code><\/pre>\n<h2>server shows<\/h2>\n<pre><code class=\"language-bash\">(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py \n * Serving Flask app 'snsread' (lazy loading)\n...\n\nincoming traffic...\n*********************\ndata\n----\n{\n    \"Message\": \"{\\\"AlarmName\\\":\\\"linux-server-errors\\\",\\\"AlarmDescription\\\":\\\"ERROR in \/var\/log\/messages\\\",\\\"AWSAccountId\\\":\\\"310843369992\\\",\\\"NewStateValue\\\":\\\"ALARM\\\",\\\"NewStateReason\\\":\\\"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24\/07\/21 01:34:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\\\",\\\"StateChangeTime\\\":\\\"2021-07-24T01:39:13.642+0000\\\",\\\"Region\\\":\\\"US West (Oregon)\\\",\\\"AlarmArn\\\":\\\"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors\\\",\\\"OldStateValue\\\":\\\"OK\\\",\\\"Trigger\\\":{\\\"MetricName\\\":\\\"messages-errors\\\",\\\"Namespace\\\":\\\"messages\\\",\\\"StatisticType\\\":\\\"Statistic\\\",\\\"Statistic\\\":\\\"MAXIMUM\\\",\\\"Unit\\\":null,\\\"Dimensions\\\":[],\\\"Period\\\":300,\\\"EvaluationPeriods\\\":1,\\\"ComparisonOperator\\\":\\\"GreaterThanThreshold\\\",\\\"Threshold\\\":0.0,\\\"TreatMissingData\\\":\\\"- TreatMissingData:                    notBreaching\\\",\\\"EvaluateLowSampleCountPercentile\\\":\\\"\\\"}}\",\n    \"MessageId\": \"7ec65853-62c5-5baf-9155-01261344a002\",\n    \"Signature\": \"mbPoUMIYpiC3DqNCft7ZgRHP9vAEyWmWhXjpeTPZxSehoB+1o4rhxWLyblugHhbJOAkZrV9sp52JIJfN2d2h7WqCXKeZxVsqqpvL1HdTWc8yCo5yWbZ\/hKibKR1A7DdXZFeyiQpnfD71sYsiFmB59lKfAi2l8f9PZDdx\/GoOboIUSoR4gwFigyEnL9E4V9C6WKb6ERXSkbwmKyMzTF82BqmsYMhXyOZXysjaqQ9Eleqh+1hv0MqUw3mPCI9IIjoHjFN7CmtrPJpf5RaYI12W1KsBUYrWI6MZQ69gwohgyvFwSRAyT9z\/z++AyMebROY3S5Fl29B+Zawfp5L44b1zzA==\",\n    \"SignatureVersion\": \"1\",\n    \"SigningCertURL\": \"https:\/\/sns.us-west-2.amazonaws.com\/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem\",\n    \"Subject\": \"ALARM: \\\"linux-server-errors\\\" in US West (Oregon)\",\n    \"Timestamp\": \"2021-07-24T01:39:13.690Z\",\n    \"TopicArn\": \"arn:aws:sns:us-west-2:310843369992:rr-events-02\",\n    \"Type\": \"Notification\",\n    \"UnsubscribeURL\": \"https:\/\/sns.us-west-2.amazonaws.com\/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c\"\n}\nheaders\n-------\nX-Amz-Sns-Message-Type: Notification\nX-Amz-Sns-Message-Id: 7ec65853-62c5-5baf-9155-01261344a002\nX-Amz-Sns-Topic-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02\nX-Amz-Sns-Subscription-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c\nContent-Length: 1901\nContent-Type: text\/plain; charset=UTF-8\nHost: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000\nConnection: Keep-Alive\nUser-Agent: Amazon Simple Notification Service Agent\nAccept-Encoding: gzip,deflate\n\njson payload\n------------\nNone\njs\n--\n{\n    \"Message\": \"{\\\"AlarmName\\\":\\\"linux-server-errors\\\",\\\"AlarmDescription\\\":\\\"ERROR in \/var\/log\/messages\\\",\\\"AWSAccountId\\\":\\\"310843369992\\\",\\\"NewStateValue\\\":\\\"ALARM\\\",\\\"NewStateReason\\\":\\\"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24\/07\/21 01:34:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\\\",\\\"StateChangeTime\\\":\\\"2021-07-24T01:39:13.642+0000\\\",\\\"Region\\\":\\\"US West (Oregon)\\\",\\\"AlarmArn\\\":\\\"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors\\\",\\\"OldStateValue\\\":\\\"OK\\\",\\\"Trigger\\\":{\\\"MetricName\\\":\\\"messages-errors\\\",\\\"Namespace\\\":\\\"messages\\\",\\\"StatisticType\\\":\\\"Statistic\\\",\\\"Statistic\\\":\\\"MAXIMUM\\\",\\\"Unit\\\":null,\\\"Dimensions\\\":[],\\\"Period\\\":300,\\\"EvaluationPeriods\\\":1,\\\"ComparisonOperator\\\":\\\"GreaterThanThreshold\\\",\\\"Threshold\\\":0.0,\\\"TreatMissingData\\\":\\\"- TreatMissingData:                    notBreaching\\\",\\\"EvaluateLowSampleCountPercentile\\\":\\\"\\\"}}\",\n    \"MessageId\": \"7ec65853-62c5-5baf-9155-01261344a002\",\n    \"Signature\": \"mbPoUMIYpiC3DqNCft7ZgRHP9vAEyWmWhXjpeTPZxSehoB+1o4rhxWLyblugHhbJOAkZrV9sp52JIJfN2d2h7WqCXKeZxVsqqpvL1HdTWc8yCo5yWbZ\/hKibKR1A7DdXZFeyiQpnfD71sYsiFmB59lKfAi2l8f9PZDdx\/GoOboIUSoR4gwFigyEnL9E4V9C6WKb6ERXSkbwmKyMzTF82BqmsYMhXyOZXysjaqQ9Eleqh+1hv0MqUw3mPCI9IIjoHjFN7CmtrPJpf5RaYI12W1KsBUYrWI6MZQ69gwohgyvFwSRAyT9z\/z++AyMebROY3S5Fl29B+Zawfp5L44b1zzA==\",\n    \"SignatureVersion\": \"1\",\n    \"SigningCertURL\": \"https:\/\/sns.us-west-2.amazonaws.com\/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem\",\n    \"Subject\": \"ALARM: \\\"linux-server-errors\\\" in US West (Oregon)\",\n    \"Timestamp\": \"2021-07-24T01:39:13.690Z\",\n    \"TopicArn\": \"arn:aws:sns:us-west-2:310843369992:rr-events-02\",\n    \"Type\": \"Notification\",\n    \"UnsubscribeURL\": \"https:\/\/sns.us-west-2.amazonaws.com\/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c\"\n}\n54.240.230.240 - - [24\/Jul\/2021 01:39:34] \"POST \/ HTTP\/1.1\" 200 -<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"<p>I wanted an easier way to test and manipulate a notification using an AWS SNS subscription. Mostly I do a<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[73,127,13],"tags":[],"class_list":["post-1837","post","type-post","status-publish","format-standard","hentry","category-aws","category-flask","category-python"],"_links":{"self":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1837","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/comments?post=1837"}],"version-history":[{"count":0,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/posts\/1837\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/media?parent=1837"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/categories?post=1837"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.ls-al.com\/wp-json\/wp\/v2\/tags?post=1837"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}