Author Archive

Sep 15

Kubernetes NodePort Load Balancing with nginx

Mostly this is done in a cloud environment where they have Kubernetes integrated with cloud load balancers and you expose kubernetes services as type LoadBalancer.

However I wanted to do this without cloud in my Virtualbox environment. Its not ideal and I wish nginx could add a port when using proxy_pass pointing to upstream.

My configuration is not ideal and does not scale well. I am using it in a POC and it is working so far so documenting for future reference.

NOTE I did not test if upstream is failing over but that is well documented for nginx so I trust it is working. You could of course change upstream mechanisms to round-robin, least-connected or ip-hash.

user www-data;
worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

http {
   map $host $serverport {
     "hello.cluster01.local"   "30000";
     "web01.cluster01.local"   "30001";
     "web02.cluster01.local"   "30002";
     default      "no_match";
   }

   upstream hello.cluster01.local-30000 {
      server 172.20.100.10:30000; 
      server 172.20.100.11:30000; 
   }

   upstream web01.cluster01.local-30001 {
      server 172.20.100.10:30001;
      server 172.20.100.11:30001;
   }

   upstream web02.cluster01.local-30002 {
      server 172.20.100.10:30002;
      server 172.20.100.11:30002;
   }

  server {
    listen 80;
    server_name "~(.*).cluster01.local";
    set $upstream $host-$serverport; 
    location / {
      proxy_set_header X-Forwarded-For $remote_addr;
      # if not load balancing pointing to one node like below is fine
      #proxy_pass http://172.20.100.10:$np;
      # with upstream you can't add a port so I have an upstream per service
      #proxy_pass http://backend:$np;
      proxy_pass http://$upstream;
      proxy_set_header Host $host;
    }
  }
}

Comments Off on Kubernetes NodePort Load Balancing with nginx
comments

Posted in Kubernetes, nginx

Jul 24

AWS SNS to http subscription receiving in python3 http server and Flask

I wanted an easier way to test and manipulate a notification using an AWS SNS subscription. Mostly I do a quick SMTP subscription to the topic. I wanted quicker and more direct feedback and also manipulate the incoming notification. I used this as reference https://gist.github.com/iMilnb/bf27da3f38272a76c801

NOTE: the code will detect if it is a subscription or notification request and route appropriately.

run a server on port 5000

$ mkdir snsread
$ cd snsread/
$ vi snsready.py
$ python3 -m venv venv
$ . venv/bin/activate

(venv) [ec2-user@ip-172-31-6-74 snsread]$ pip3 install Flask
...

(venv) [ec2-user@ip-172-31-6-74 snsread]$ pip3 install requests
...

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.31.6.74:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 729-981-989

test using curl from public client

$ curl -I ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 3
Server: Werkzeug/2.0.1 Python/3.7.10
Date: Fri, 23 Jul 2021 21:38:20 GMT

when testing curl request python server shows

99.122.138.75 - - [23/Jul/2021 21:38:20] "HEAD / HTTP/1.1" 200 -

when testing publish direct from SNS topic the python server shows

205.251.234.35 - - [23/Jul/2021 21:41:26] "POST / HTTP/1.1" 200 -

Add subscription in topic rr-events-02 as http://ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000

server shows during subscription

205.251.234.35 - - [23/Jul/2021 21:41:26] "POST / HTTP/1.1" 200 -

topic > publish message

server shows

raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

NOTE: reference code was not exactly matching my specific notifications so need some tweaking  as well as some json.loads(json.dumps())) love.

initial successful from actual Cloudwatch alarm sent to SNS

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
...
incoming...
headers: 
json payload: 
js: {"AlarmName":"linux-server-errors","AlarmDescription":"ERROR in /var/log/messages","AWSAccountId":"310843369992","NewStateValue":"ALARM","NewStateReason":"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 00:29:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).","StateChangeTime":"2021-07-24T00:34:13.630+0000","Region":"US West (Oregon)","AlarmArn":"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors","OldStateValue":"OK","Trigger":{"MetricName":"messages-errors","Namespace":"messages","StatisticType":"Statistic","Statistic":"MAXIMUM","Unit":null,"Dimensions":[],"Period":300,"EvaluationPeriods":1,"ComparisonOperator":"GreaterThanThreshold","Threshold":0.0,"TreatMissingData":"- TreatMissingData:                    notBreaching","EvaluateLowSampleCountPercentile":""}}
205.251.233.161 - - [24/Jul/2021 00:34:13] "POST / HTTP/1.1" 200 -

Publish message directly from topic in console

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.31.6.74:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 729-981-989

incoming traffic...
*********************
data
----
{
    "Message": "my raw msg...",
    "MessageId": "149d65b0-9a32-524a-95ac-3cc5ea934e04",
    "Signature": "JZp1x1mOXw2PjwhIFfA4QmNc74pzai5G3kbXyYvnNW1a5YkexGKSCpmYLT/LEFqxfJy6VFYDGmb/+Ty2aQO0qQlO2wd5D+SkZOHjNAs0u+lCuw+cOBYCtyRAWJI3c5zGR928WE4PuWEoNgg8NQnW9RBRkCEqcEgQChjgbZlxs2ehvl1LZ/1rkcWzG3+/p5wZL0czhkRA2dx5JeM7d2zCuFisp+2rQN6aRfRObV0YcBqBVFwUmL2C7uxgPt6TTf4nfpgFqDKrV6S/BfOJqWTNKDkUKvUQCk5inxOOOpFmDs2V6LhkV1kRGgXAx5moQTWTTAc/CC+1N8ylXyUdES4fAA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "my msg",
    "Timestamp": "2021-07-24T01:35:29.357Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
headers
-------
X-Amz-Sns-Message-Type: Notification
X-Amz-Sns-Message-Id: 149d65b0-9a32-524a-95ac-3cc5ea934e04
X-Amz-Sns-Topic-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02
X-Amz-Sns-Subscription-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c
Content-Length: 946
Content-Type: text/plain; charset=UTF-8
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate

json payload
------------
None
js
--
{
    "Message": "my raw msg...",
    "MessageId": "149d65b0-9a32-524a-95ac-3cc5ea934e04",
    "Signature": "JZp1x1mOXw2PjwhIFfA4QmNc74pzai5G3kbXyYvnNW1a5YkexGKSCpmYLT/LEFqxfJy6VFYDGmb/+Ty2aQO0qQlO2wd5D+SkZOHjNAs0u+lCuw+cOBYCtyRAWJI3c5zGR928WE4PuWEoNgg8NQnW9RBRkCEqcEgQChjgbZlxs2ehvl1LZ/1rkcWzG3+/p5wZL0czhkRA2dx5JeM7d2zCuFisp+2rQN6aRfRObV0YcBqBVFwUmL2C7uxgPt6TTf4nfpgFqDKrV6S/BfOJqWTNKDkUKvUQCk5inxOOOpFmDs2V6LhkV1kRGgXAx5moQTWTTAc/CC+1N8ylXyUdES4fAA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "my msg",
    "Timestamp": "2021-07-24T01:35:29.357Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
54.240.230.187 - - [24/Jul/2021 01:36:11] "POST / HTTP/1.1" 200 -

from linux server custom json

[rrosso@fedora ~]$ curl -i -H "Content-Type: application/json" -X POST -d '{"userId":"1", "username": "fizz bizz"}' http://ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 3
Server: Werkzeug/2.0.1 Python/3.7.10
Date: Sat, 24 Jul 2021 00:59:14 GMT

OK

server shows

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
....

incoming traffic...
data: 
{'userId': '1', 'username': 'fizz bizz'}
headers: 
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
User-Agent: curl/7.76.1
Accept: */*
Content-Type: application/json
Content-Length: 39

json payload: {'userId': '1', 'username': 'fizz bizz'}
99.122.138.75 - - [24/Jul/2021 00:55:44] "POST / HTTP/1.1" 200 -

initial successful from actual Cloudwatch alarm sent to SNS. injected to server messages with logger

[root@ip-172-31-6-74 ~]# logger "ERROR: WTF6 is going on..."

server shows

(venv) [ec2-user@ip-172-31-6-74 snsread]$ python3 snsread.py 
 * Serving Flask app 'snsread' (lazy loading)
...

incoming traffic...
*********************
data
----
{
    "Message": "{\"AlarmName\":\"linux-server-errors\",\"AlarmDescription\":\"ERROR in /var/log/messages\",\"AWSAccountId\":\"310843369992\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 01:34:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2021-07-24T01:39:13.642+0000\",\"Region\":\"US West (Oregon)\",\"AlarmArn\":\"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"messages-errors\",\"Namespace\":\"messages\",\"StatisticType\":\"Statistic\",\"Statistic\":\"MAXIMUM\",\"Unit\":null,\"Dimensions\":[],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanThreshold\",\"Threshold\":0.0,\"TreatMissingData\":\"- TreatMissingData:                    notBreaching\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
    "MessageId": "7ec65853-62c5-5baf-9155-01261344a002",
    "Signature": "mbPoUMIYpiC3DqNCft7ZgRHP9vAEyWmWhXjpeTPZxSehoB+1o4rhxWLyblugHhbJOAkZrV9sp52JIJfN2d2h7WqCXKeZxVsqqpvL1HdTWc8yCo5yWbZ/hKibKR1A7DdXZFeyiQpnfD71sYsiFmB59lKfAi2l8f9PZDdx/GoOboIUSoR4gwFigyEnL9E4V9C6WKb6ERXSkbwmKyMzTF82BqmsYMhXyOZXysjaqQ9Eleqh+1hv0MqUw3mPCI9IIjoHjFN7CmtrPJpf5RaYI12W1KsBUYrWI6MZQ69gwohgyvFwSRAyT9z/z++AyMebROY3S5Fl29B+Zawfp5L44b1zzA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "ALARM: \"linux-server-errors\" in US West (Oregon)",
    "Timestamp": "2021-07-24T01:39:13.690Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
headers
-------
X-Amz-Sns-Message-Type: Notification
X-Amz-Sns-Message-Id: 7ec65853-62c5-5baf-9155-01261344a002
X-Amz-Sns-Topic-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02
X-Amz-Sns-Subscription-Arn: arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c
Content-Length: 1901
Content-Type: text/plain; charset=UTF-8
Host: ec2-54-189-23-28.us-west-2.compute.amazonaws.com:5000
Connection: Keep-Alive
User-Agent: Amazon Simple Notification Service Agent
Accept-Encoding: gzip,deflate

json payload
------------
None
js
--
{
    "Message": "{\"AlarmName\":\"linux-server-errors\",\"AlarmDescription\":\"ERROR in /var/log/messages\",\"AWSAccountId\":\"310843369992\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [1.0 (24/07/21 01:34:00)] was greater than the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2021-07-24T01:39:13.642+0000\",\"Region\":\"US West (Oregon)\",\"AlarmArn\":\"arn:aws:cloudwatch:us-west-2:310843369992:alarm:linux-server-errors\",\"OldStateValue\":\"OK\",\"Trigger\":{\"MetricName\":\"messages-errors\",\"Namespace\":\"messages\",\"StatisticType\":\"Statistic\",\"Statistic\":\"MAXIMUM\",\"Unit\":null,\"Dimensions\":[],\"Period\":300,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanThreshold\",\"Threshold\":0.0,\"TreatMissingData\":\"- TreatMissingData:                    notBreaching\",\"EvaluateLowSampleCountPercentile\":\"\"}}",
    "MessageId": "7ec65853-62c5-5baf-9155-01261344a002",
    "Signature": "mbPoUMIYpiC3DqNCft7ZgRHP9vAEyWmWhXjpeTPZxSehoB+1o4rhxWLyblugHhbJOAkZrV9sp52JIJfN2d2h7WqCXKeZxVsqqpvL1HdTWc8yCo5yWbZ/hKibKR1A7DdXZFeyiQpnfD71sYsiFmB59lKfAi2l8f9PZDdx/GoOboIUSoR4gwFigyEnL9E4V9C6WKb6ERXSkbwmKyMzTF82BqmsYMhXyOZXysjaqQ9Eleqh+1hv0MqUw3mPCI9IIjoHjFN7CmtrPJpf5RaYI12W1KsBUYrWI6MZQ69gwohgyvFwSRAyT9z/z++AyMebROY3S5Fl29B+Zawfp5L44b1zzA==",
    "SignatureVersion": "1",
    "SigningCertURL": "https://sns.us-west-2.amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
    "Subject": "ALARM: \"linux-server-errors\" in US West (Oregon)",
    "Timestamp": "2021-07-24T01:39:13.690Z",
    "TopicArn": "arn:aws:sns:us-west-2:310843369992:rr-events-02",
    "Type": "Notification",
    "UnsubscribeURL": "https://sns.us-west-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-west-2:310843369992:rr-events-02:a0fbe74a-a10a-4d85-a405-e0627a7e075c"
}
54.240.230.240 - - [24/Jul/2021 01:39:34] "POST / HTTP/1.1" 200 -

Comments Off on AWS SNS to http subscription receiving in python3 http server and Flask
comments

Posted in AWS, Flask, Python

Apr 29

Webm Video Clips

Now and then I create a short clip from a video. Mostly mp4 video. I use ffmpeg and cut out the clip based on start and end time. Linux mostly does not play mp4 out of the box and need a few packages. I started using webm which so far have better compatibility playing with Gnome standard video player, VLC (whihc always plays everything anyhow), mobile phone messaging apps etc.. For my convenience here is the detail from a script I use.

➜  scripts cat video_to_webm.sh 
#!/bin/bash
VIDEOS=~/Downloads/video-to-webm/
find "$VIDEOS" -name '*.mp4' -exec sh -c 'ffmpeg -i "$0" -c:v libvpx -crf 10 -b:v 1M -c:a libvorbis "${0%%.mp4}.webm"' {} \;
exit;

Example of a creating a clip.

$ ffmpeg -ss 1:04:42 -to 1:07:38 -i The\ Podcast\ of\ the\ Lotus\ Eaters\ \#120.mp4 -c:v libvpx -crf 10 -b:v 1M -c:a libvorbis The_Podcast_of_the_Lotus_Eaters_120-keto_fascism.webm

Comments Off on Webm Video Clips
comments

Posted in Video Encoding

Apr 17

Token Balance Decimals

Balance Decimals

Noting my python code for reference. If familiar with crypto currency tokens you may find the token balance does not have decimals but rather store the decimal value separately.

For example this query shows the balance:

$ curl -s https://api.ethplorer.io/getAddressInfo/0xbcB79558e0d66475882A36FaF4124Ec45aA70dA3\?apiKey\=freekey | jq -r '.tokens[0].balance' 
1001193304561787500000

If you look at the token detail you see the decimals recorded:

$ curl -s https://api.ethplorer.io/getAddressInfo/0xbcB79558e0d66475882A36FaF4124Ec45aA70dA3\?apiKey\=freekey | jq -r '.tokens[0].tokenInfo.decimals'
18

More on the decimals here:
The Technology Behind Ethereum Tokens

For my python3 code I used as follow:

$ python3                                                                                                                         
Python 3.8.6 (default, Jan 27 2021, 15:42:20) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 1001193304561787500000*10**-18
1001.1933045617875

Comments Off on Token Balance Decimals
comments

Posted in Crypto, Python

Apr 08

Gnome Desktop Shortcut

GNOME Desktop Shortcut

Just another note about this. See previous also.

Create in user Desktop folder

$ cat crypto-report.desktop 
[Desktop Entry]
Version=1.0
Exec=/home/rrosso/scripts/crypto-report.sh
Name=crypto-report
GenericName=crypto-report
Comment=run crypto-report script
Encoding=UTF-8
Terminal=true
Type=Application
Categories=Application;Network;

Script

$ cat ~/scripts/crypto-report.sh 
#!/bin/bash

commands () {
    cd /TANK/DATA/MySrc/crypto-totals-api
    python3 consolidated-crypto-report.py
    $SHELL # keep the terminal open after the previous commands are executed
}

export -f commands

gnome-terminal --profile=crypto-report -- bash -c "commands"

NOTE: May have to right click on shortcut on the desktup and allow to tun.

Comments Off on Gnome Desktop Shortcut
comments

Posted in Gnome

Mar 24

Go Import JSON File Into MongoDB Unstructured

During a recent POC I imported a Terraform state file(JSON) into MongoDB using golang. This may not exactly fit any use case since it is simpler and use actual file(s) and not MongoDB to store a Terraform state file, but this helps record my effort as well as the unstructured nature of the JSON ie I am not defining a structure just using an interface{}.

Insert From File

package main

import (
    "context"
    "fmt"
    "os"
    "io/ioutil"
    "encoding/json"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

func main() {
    clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
    client, e := mongo.Connect(context.TODO(), clientOptions)
    CheckError(e)
    collection := client.Database("tfstatedb").Collection("states")
    jsonFile, err := os.Open("terraform.tfstate")
    if err != nil {
        fmt.Println(err)
    }
    defer jsonFile.Close()

    byteValue, _ := ioutil.ReadAll(jsonFile)
    var result map[string]interface{}
    json.Unmarshal([]byte(byteValue), &result) 
    _, e = collection.InsertOne(context.TODO(), result)
    CheckError(e)
}

func CheckError(e error) {
    if e != nil {
        fmt.Println(e)
    }
}

FindOne From DB

package main

import (
    "flag"
    "log"
    "context"
    "fmt"
    "encoding/json"
    "go.mongodb.org/mongo-driver/bson"
    "go.mongodb.org/mongo-driver/bson/primitive"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/options"
)

var (
     flagId string
)

func init() {
     flag.StringVar(&flagId, "id", "", "mongodb ObjectId")
}

func PrettyPrint(v interface{}) (err error) {
      b, err := json.MarshalIndent(v, "", "  ")
      if err == nil {
         fmt.Println(string(b))
      }
      return
}

func main() {
    if !flag.Parsed() {
    flag.Parse()
    }
    if flagId != "" {
          fmt.Println("finding id: " + flagId)
    } else {
          log.Fatal("no _id specified")
    }

    clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
    client, e := mongo.Connect(context.TODO(), clientOptions)
    CheckError(e)

    e = client.Ping(context.TODO(), nil)
    CheckError(e)

    collection := client.Database("tfstatedb").Collection("states")

    var res interface{}

    docID, e := primitive.ObjectIDFromHex(flagId)
    CheckError(e)

    filter := bson.M{"_id": docID}

    tempResult := bson.M{}
    e = collection.FindOne(context.TODO(), filter).Decode(&tempResult)
    if e == nil {   
      obj, _ := json.Marshal(tempResult)
      e = json.Unmarshal(obj, &res)
    }
    PrettyPrint(res)
}

func CheckError(e error) {
    if e != nil {
        fmt.Println(e)
    }
}

Comments Off on Go Import JSON File Into MongoDB Unstructured
comments

Posted in go, MongoDB, Terraform

Mar 22

Terraform Stateserver Using Python

Terraform can utilize a http backend for maintaining state. This is a test of a Terraform http backend using a server implemented with python.

NOTE: checked source into https://github.com/rrossouw01/terraform-stateserver-py/

recipe and components

Using Virtualbox Ubuntu 20.10 and followed links:

setup

$ mkdir tf-state-server
$ cd tf-state-server

$ virtualenv -p python3 venv
created virtual environment CPython3.8.6.final.0-64 in 174ms
  creator CPython3Posix(dest=/home/rrosso/tf-state-server/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/rrosso/.local/share/virtualenv)
    added seed packages: pip==20.1.1, pkg_resources==0.0.0, setuptools==44.0.0, wheel==0.34.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

$ source venv/bin/activate

(venv) $ pip install -U -r requirements.txt
Collecting flask
  Using cached Flask-1.1.2-py2.py3-none-any.whl (94 kB)
Collecting flask_restful
  Downloading Flask_RESTful-0.3.8-py2.py3-none-any.whl (25 kB)
Collecting itsdangerous>=0.24
  Using cached itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting Werkzeug>=0.15
  Using cached Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting Jinja2>=2.10.1
  Using cached Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
Collecting click>=5.1
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting pytz
  Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
     |████████████████████████████████| 510 kB 3.0 MB/s 
Collecting six>=1.3.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting aniso8601>=0.82
  Downloading aniso8601-9.0.1-py2.py3-none-any.whl (52 kB)
     |████████████████████████████████| 52 kB 524 kB/s 
Collecting MarkupSafe>=0.23
  Using cached MarkupSafe-1.1.1-cp38-cp38-manylinux2010_x86_64.whl (32 kB)
Installing collected packages: itsdangerous, Werkzeug, MarkupSafe, Jinja2, click, flask, pytz, six, aniso8601, flask-restful
Successfully installed Jinja2-2.11.3 MarkupSafe-1.1.1 Werkzeug-1.0.1 aniso8601-9.0.1 click-7.1.2 flask-1.1.2 flask-restful-0.3.8 itsdangerous-1.1.0 pytz-2021.1 six-1.15.0

(venv) $ python3 stateserver.py
 * Serving Flask app "stateserver" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://192.168.1.235:5000/ (Press CTRL+C to quit)
...

terraform point to remote http

➜  cat main.tf 
terraform {
  backend "http" {
    address = "http://192.168.1.235:5000/terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a"
    lock_address = "http://192.168.1.235:5000/terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a"
    lock_method = "PUT"
    unlock_address = "http://192.168.1.235:5000/terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a"
    unlock_method = "DELETE"
  }
}

➜  source ../env-vars 

➜  terraform init    

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "http" backend. No existing state was found in the newly
  configured "http" backend. Do you want to copy this state to the new "http"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.oci: version = "~> 4.17"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

server shows
---
````bash
...
192.168.1.111 - - [16/Mar/2021 10:50:00] "POST /terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a?ID=84916e49-1b44-1b32-2058-62f28e1e8ee7 HTTP/1.1" 200 -
192.168.1.111 - - [16/Mar/2021 10:50:00] "DELETE /terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 -
192.168.1.111 - - [16/Mar/2021 10:50:00] "GET /terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 -
...

$ ls -la .stateserver/
total 24
drwxrwxr-x 2 rrosso rrosso 4096 Mar 16 10:50 .
drwxrwxr-x 4 rrosso rrosso 4096 Mar 16 10:43 ..
-rw-rw-r-- 1 rrosso rrosso 4407 Mar 16 10:50 4cdd0c76-d78b-11e9-9bea-db9cd8374f3a
-rw-rw-r-- 1 rrosso rrosso 5420 Mar 16 10:50 4cdd0c76-d78b-11e9-9bea-db9cd8374f3a.log

$ more .stateserver/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a*
::::::::::::::
.stateserver/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a
::::::::::::::
{
    "lineage": "9b756fb7-e41a-7cd6-d195-d794f377e7be",
    "outputs": {},
    "resources": [
{
"instances": [
{
"attributes": {
"compartment_id": null,
"id": "ObjectStorageNamespaceDataSource-0",
"namespace": "axwscg6apasa"
},
"schema_version": 0
}
],
...
    "serial": 0,
    "terraform_version": "0.12.28",
    "version": 4
}
::::::::::::::
.stateserver/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a.log
::::::::::::::
lock: {
    "Created": "2021-03-16T15:49:34.567178267Z",
    "ID": "7eea0d70-5f53-e475-041b-bcc393f4a92d",
    "Info": "",
    "Operation": "migration destination state",
    "Path": "",
    "Version": "0.12.28",
    "Who": "rrosso@desktop01"
}
unlock: {
    "Created": "2021-03-16T15:49:34.567178267Z",
    "ID": "7eea0d70-5f53-e475-041b-bcc393f4a92d",
    "Info": "",
    "Operation": "migration destination state",
    "Path": "",
    "Version": "0.12.28",
    "Who": "rrosso@desktop01"
}
lock: {
    "Created": "2021-03-16T15:49:45.760917508Z",
    "ID": "84916e49-1b44-1b32-2058-62f28e1e8ee7",
    "Info": "",
    "Operation": "migration destination state",
    "Path": "",
    "Version": "0.12.28",
    "Who": "rrosso@desktop01"
}
state_write: {
    "lineage": "9b756fb7-e41a-7cd6-d195-d794f377e7be",
    "outputs": {},
    "resources": [
{
"instances": [
{
"attributes": {
"compartment_id": null,
"id": "ObjectStorageNamespaceDataSource-0",
"namespace": "axwscg6apasa"
},
"schema_version": 0
}
...

stateserver shows during plan

...
192.168.1.111 - - [16/Mar/2021 10:54:12] "PUT /terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 -
192.168.1.111 - - [16/Mar/2021 10:54:12] "GET /terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 -
192.168.1.111 - - [16/Mar/2021 10:54:15] "DELETE /terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 -
...

source

$ more requirements.txt stateserver.py 
::::::::::::::
requirements.txt
::::::::::::::
flask
flask_restful
::::::::::::::
stateserver.py
::::::::::::::
#!/usr/bin/python3

import flask
import flask_restful
import json
import logging
import os

app = flask.Flask(__name__)
api = flask_restful.Api(app)

@app.before_request
def log_request_info():
    headers = []
    for header in flask.request.headers:
        headers.append('%s = %s' % (header[0], header[1]))

    body = flask.request.get_data().decode('utf-8').split('\n')

    app.logger.debug(('%(method)s for %(url)s...\n'
                      '    Header -- %(headers)s\n'
                      '    Body -- %(body)s\n')
                     % {
        'method': flask.request.method,
        'url': flask.request.url,
        'headers': '\n    Header -- '.join(headers),
        'body': '\n           '.join(body)
    })

class Root(flask_restful.Resource):
    def get(self):
        resp = flask.Response(
            'Oh, hello',
            mimetype='text/html')
        resp.status_code = 200
        return resp

class StateStore(object):
    def __init__(self, path):
        self.path = path
        os.makedirs(self.path, exist_ok=True)

    def _log(self, id, op, data):
        log_file = os.path.join(self.path, id) + '.log'
        with open(log_file, 'a') as f:
            f.write('%s: %s\n' %(op, data))

    def get(self, id):
        file = os.path.join(self.path, id)
        if os.path.exists(file):
            with open(file) as f:
                d = f.read()
                self._log(id, 'state_read', {})
                return json.loads(d)
        return None

    def put(self, id, info):
        file = os.path.join(self.path, id)
        data = json.dumps(info, indent=4, sort_keys=True)
        with open(file, 'w') as f:
            f.write(data)
            self._log(id, 'state_write', data)

    def lock(self, id, info):
        # NOTE(mikal): this is racy, but just a demo
        lock_file = os.path.join(self.path, id) + '.lock'
        if os.path.exists(lock_file):
            # If the lock exists, it should be a JSON dump of information about
            # the lock holder
            with open(lock_file) as f:
                l = json.loads(f.read())
            return False, l

        data = json.dumps(info, indent=4, sort_keys=True)
        with open(lock_file, 'w') as f:
            f.write(data)
        self._log(id, 'lock', data)
        return True, {}

    def unlock(self, id, info):
        lock_file = os.path.join(self.path, id) + '.lock'
        if os.path.exists(lock_file):
            os.unlink(lock_file)
            self._log(id, 'unlock', json.dumps(info, indent=4, sort_keys=True))
            return True
        return False

state = StateStore('.stateserver')

class TerraformState(flask_restful.Resource):
    def get(self, tf_id):
        s = state.get(tf_id)
        if not s:
            flask.abort(404)
        return s

    def post(self, tf_id):
        print(flask.request.form)
        s = state.put(tf_id, flask.request.json)
        return {}

class TerraformLock(flask_restful.Resource):
    def put(self, tf_id):
        success, info = state.lock(tf_id, flask.request.json)
        if not success:
            flask.abort(423, info)
        return info

    def delete(self, tf_id):
        if not state.unlock(tf_id, flask.request.json):
            flask.abort(404)
        return {}

api.add_resource(Root, '/')
api.add_resource(TerraformState, '/terraform_state/')
api.add_resource(TerraformLock, '/terraform_lock/')

if __name__ == '__main__':
    # Note this is not run with the flask task runner...
    app.log = logging.getLogger('werkzeug')
    app.log.setLevel(logging.DEBUG)
    #app.run(host='0.0.0.0', debug=True)

Comments Off on Terraform Stateserver Using Python
comments

Posted in Python, Terraform

Mar 22

Terraform Stateserver Using Go

Terraform can utilize a http backend for maintaining state. This is a test of a Terraform http backend using a server implemented with go.

recipe and components

• Using Virtualbox Ubuntu 20.10 and links listed below
• Run as http server
• Change to https
• For a future test terraform http backend locking feature and/or include locking in server

links

• https://www.terraform.io/docs/language/settings/backends/http.html
• https://linuxhint.com/install-golang-ubuntu/
• https://github.com/MerlinDMC/go-terraform-stateserver
• https://github.com/denji/golang-tls

setup go and test

# go version
go version go1.14.7 linux/amd64

$ pwd
/home/rrosso/tf-go-server
$ cat main.go 
package main
import "fmt"
func main() {
fmt.Println("Hello World");
}
$ go run main.go 
Hello World

$ go build main.go 
$ ./main 
Hello World

run server and test response

$ go run tf-stateserver.go -data_path=/home/rrosso/tf-go-server -listen_address=192.168.1.235:8080

$ curl -sL http://192.168.1.235:8080 | xxd
00000000: 3430 3420 7061 6765 206e 6f74 2066 6f75  404 page not fou
00000010: 6e64 0and.

terraform init with http backend

➜  terraform-poc tail -8 main.tf 
#}
## POC to test a go server
terraform {
  backend "http" {
    address = "http://192.168.1.235:8080/states/terraform.tfstate"
  }
}

➜  terraform-poc terraform init

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "http" backend. No existing state was found in the newly
  configured "http" backend. Do you want to copy this state to the new "http"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.oci: version = "~> 4.17"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

observe server log messages

• file not found if no file exist during init is expected
• if file same and already there GET fine during init
• plan does a GET
• apply does a POST
• did not observe a DELETE
$ go run tf-stateserver.go -data_path=/home/rrosso/tf-go-server -listen_address=192.168.1.235:8080
2021/03/18 09:57:41 received GET for state file: /home/rrosso/tf-go-server/states/terraform.tfstate
...
2021/03/18 09:58:33 received POST for state file: /home/rrosso/tf-go-server/states/terraform.tfstate
...
2021/03/18 10:00:50 received GET for state file: /home/rrosso/tf-go-server/states/terraform.tfstate
2021/03/18 10:00:50 during GET cannot open file: open /home/rrosso/tf-go-server/states/terraform.tfstate: no such file or directory
2021/03/18 10:00:50 received GET for state file: /home/rrosso/tf-go-server/states/terraform.tfstate

change to self signed cert for https test

NOTE:
• use sudo since linux instance is not running low ports like 443 for normal users
• need terraform skip_cert_verification for this self signed certificate

$ openssl genrsa -out server.key 2048
Generating RSA private key, 2048 bit long modulus (2 primes)
..........................+++++
.......................................................+++++
e is 65537 (0x010001)

$ openssl ecparam -genkey -name secp384r1 -out server.key

$ openssl req -new -x509 -sha256 -key server.key -out server.crt -days 3650
...

$ sudo go run tf-stateserver.go -certfile="server.crt" -keyfile="server.key" -data_path=./ -listen_address=192.168.1.235:443

$ curl -sL http://192.168.1.235:443 | xxd
00000000: 436c 6965 6e74 2073 656e 7420 616e 2048  Client sent an H
00000010: 5454 5020 7265 7175 6573 7420 746f 2061  TTP request to a
00000020: 6e20 4854 5450 5320 7365 7276 6572 2e0a  n HTTPS server..

➜  terraform-poc tail -8 main.tf 
#}
## POC to test a go server
terraform {
  backend "http" {
    address = "https://192.168.1.235/states/terraform.tfstate"
    skip_cert_verification = true
  }
}

NOTE: terraform init, plan, apply works

additional links

• https://appdividend.com/2019/11/29/golang-json-example-how-to-use-json-with-go/
• https://www.golangprograms.com/web-application-to-read-and-write-json-data-in-json-file.html
• https://tutorialedge.net/golang/parsing-json-with-golang/
• https://dzone.com/articles/how-to-write-a-http-rest-api-server-in-go-in-minut

source

NOTE: subsequently checked source into github here: https://github.com/rrossouw01/terraform-stateserver-go/blob/main/server.go

Can use the file directly with simple wget to the raw link for example:

$ wget https://raw.githubusercontent.com/rrossouw01/terraform-stateserver-go/main/server.go
$ cat tf-stateserver.go 
package main
//https://github.com/MerlinDMC/go-terraform-stateserver/blob/master/main.go

import (
    "flag"
    "io"
    "log"
    "net/http"
    "os"
    "path/filepath"
)

var (
    flagDataPath string
    flagCertFile string
    flagKeyFile  string

    flagListenAddress string
)

func init() {
    flag.StringVar(&flagCertFile, "certfile", "", "path to the certs file for https")
    flag.StringVar(&flagKeyFile, "keyfile", "", "path to the keyfile for https")
    flag.StringVar(&flagDataPath, "data_path", os.TempDir(), "path to the data storage directory")

    flag.StringVar(&flagListenAddress, "listen_address", "0.0.0.0:8080", "address:port to bind listener on")
    //flag.StringVar(&flagListenAddress, "listen_address", "192.168.1.235:8080", "address:port to bind listener on")
}

func main() {
    if !flag.Parsed() {
        flag.Parse()
    }

    router := http.NewServeMux()
    router.HandleFunc("/", requestHandler)

    if flagCertFile != "" && flagKeyFile != "" {
          log.Fatal(http.ListenAndServeTLS(flagListenAddress, flagCertFile, flagKeyFile, router))
    } else {
          log.Fatal(http.ListenAndServe(flagListenAddress, router))
    }
}

func requestHandler(res http.ResponseWriter, req *http.Request) {
    if req.URL.Path == "/" {
        http.NotFound(res, req)
        return
    }

    stateStorageFile := filepath.Join(flagDataPath, req.URL.Path)
    stateStorageDir := filepath.Dir(stateStorageFile)

    switch req.Method {
    case "GET":
        log.Printf("received GET for state file: %s\n", stateStorageFile)
        fh, err := os.Open(stateStorageFile)
        if err != nil {
            log.Printf("during GET cannot open file: %s\n", err)
            goto not_found
        }
        defer fh.Close()

        res.WriteHeader(200)

        io.Copy(res, fh)

        return
    case "POST":
        log.Printf("received POST for state file: %s\n", stateStorageFile)
        if err := os.MkdirAll(stateStorageDir, 0750); err != nil && !os.IsExist(err) {
            log.Printf("during POST cannot create parent directories: %s\n", err)
            goto not_found
        }

        fh, err := os.OpenFile(stateStorageFile, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666)
        if err != nil {
            log.Printf("cannot open file: %s\n", err)
            goto not_found
        }
        defer fh.Close()

        if _, err := io.Copy(fh, req.Body); err != nil {
            log.Printf("failed to stream data into statefile: %s\n", err)
            goto not_found
        }

        res.WriteHeader(200)

        return
    case "DELETE":
        log.Printf("received DELETE for state file: %s\n", stateStorageFile)
        //if os.RemoveAll(stateStorageFile) != nil {
        if err := os.RemoveAll(stateStorageFile); err != nil {
            log.Printf("cannot remove file: %s\n", err)
            goto not_found
        }

        res.WriteHeader(200)

        return
    }

not_found:
    http.NotFound(res, req)
}

Comments Off on Terraform Stateserver Using Go
comments

Posted in go, Terraform

Jan 30

Python Flask API Using MongoDB

Python Flask API Using MongoDB

After a recent experiment I did to capture the output of jobs with json output from multiple server's to a central collector, I kept my notes for future reference. In short I used my restic backup json format to submit into MOngoDB via a little custom API.

The notes are showing scripts I used whihc of course may not work in your environment they are for reference only.

Setup

  • ubuntu 20.10 virtualbox host (use my snapshot called pre-setup)
  • run setup.sh from ~/mongodb-api
  • run python3 create-db.py from ~/mongodb-api
  • run python3 apiserver.py from ~/mongodb-api
  • test GET and POST with postman from my desktop

source code

$ more setup.sh create-db.py apiserver.py 
::::::::::::::
setup.sh
::::::::::::::
#!/bin/bash
sudo apt install -y gnupg python3-pip python3-flask python3-flask-mongoengine python3-virtualenv
sudo pip install flask pymongo flask_pymongo
sudo snap install robo3t-snap

wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
sudo apt-get update
sudo apt-get install -y mongodb-org
ps --no-headers -o comm 1
sudo systemctl start mongod
sudo systemctl status mongod
sudo systemctl enable mongod

cd mongodb-api
virtualenv -p python3 venv
source venv/bin/activate

## run in a new terminal one time
#python3 create-db.py

## run in this terminal after db created
#python3 apiserver.py

::::::::::::::
create-db.py
::::::::::::::
# create-db.py

from pymongo import MongoClient

client = MongoClient('mongodb://localhost:27017/')

mydb = client['restic-db']

import datetime
## example json output from a restic backup run
#{"message_type":"summary","files_new":2,"files_changed":1,"files_unmodified":205021,"dirs_new":0,"dirs_changed":6,"dirs_unmodified":28000,"data_blobs":3,"tree_blobs":7,"data_added":17802,"total_files_processed":205024,"total_bytes_processed":6002872966,"total_duration":29.527921058,"snapshot_id":"84de0f00"

myrecord = {
        "message_type": "summary",
    "files_new":2,
    "files_changed":1,
    "files_unmodified":205021,
    "dirs_new":0,
    "dirs_changed":6,
    "dirs_unmodified":28000,
    "data_blobs":3,
    "tree_blobs":7,
    "data_added":17802,
    "total_files_processed":205024,
    "total_bytes_processed":6002872966,
    "total_duration":29.527921058,
    "snapshot_id":"84de0f00",
        "tags" : ["prod", "daily", "weekly"],
        "date_inserted" : datetime.datetime.utcnow(),
        "hostname" : "desktop01"
        }

record_id = mydb.logs.insert_one(myrecord)

print (record_id)
print (mydb.list_collection_names())
::::::::::::::
apiserver.py
::::::::::::::
# mongo.py

from flask import Flask
from flask import jsonify
from flask import request
from flask_pymongo import PyMongo

app = Flask(__name__)

app.config['MONGO_DBNAME'] = 'restic-db'
app.config['MONGO_URI'] = 'mongodb://localhost:27017/restic-db'

mongo = PyMongo(app)

@app.route('/')
def index():
  return jsonify({'result' : 'v0.9.0'})

@app.route('/entry', methods=['GET'])
def get_all_entries():
  entry = mongo.db.logs
  output = []
  for s in entry.find():
    output.append({'hostname' : s['hostname'], 'message_type' : s['message_type']})
  return jsonify({'result' : output})

@app.route('/entry/', methods=['GET'])
def get_one_entry(hostname):
  entry = mongo.db.logs
  s = entry.find_one({'hostname' : hostname})
  if s:
    output = {'hostname' : s['hostname'], 'message_type' : s['message_type']}
  else:
    output = "No such hostname"
  return jsonify({'result' : output})

@app.route('/entry', methods=['POST'])
def add_entry():
  entry = mongo.db.logs
  #details = request.get_json()
  #hostname = details["hostname"]
  hostname = request.json['hostname']
  #hostname = "'" + str(request.values.get("hostname")) + "'"
  message_type = request.json['message_type']
  #message_type = "'" + str(request.values.get("message_type")) + "'"
  #message_type = details.message_type

  import datetime

 ## example json output from a restic backup run #{"message_type":"summary","files_new":2,"files_changed":1,"files_unmodified":205021,"dirs_new":0,"dirs_changed":6,"dirs_unmodified":28000,"data_blobs":3,"tree_blobs":7,"data_added":17802,"total_files_processed":205024,"total_bytes_processed":6002872966,"total_duration":29.527921058,"snapshot_id":"84de0f00"}

  myrecord = {
        "message_type": message_type,
        "files_new":2,
        "files_changed":1,
        "files_unmodified":205021,
        "dirs_new":0,
        "dirs_changed":6,
        "dirs_unmodified":28000,
        "data_blobs":3,
        "tree_blobs":7,
        "data_added":17802,
        "total_files_processed":205024,
        "total_bytes_processed":6002872966,
        "total_duration":29.527921058,
        "snapshot_id":"84de0f00",
        "tags" : ["prod", "daily", "weekly"],
        "date_inserted" : datetime.datetime.utcnow(),
        "hostname" : hostname
        }

  #entry_id = entry.insert_one({'hostname': hostname, 'message_type': message_type})
  entry_id = entry.insert_one(myrecord)

  new_entry = entry.find_one({'_id': entry_id.inserted_id })
  output = {'hostname' : new_entry['hostname'], 'message_type' : new_entry['message_type']}
  return jsonify({'result' : output})

if __name__ == '__main__':
    #app.run(debug=True)
    app.run (host = "192.168.1.235", port = 5000)

Test API's from my desktop using Postman

Comments Off on Python Flask API Using MongoDB
comments

Posted in API, Flask, MongoDB, Python

Jan 17

Restic find and list one file

Sometimes I just want to quickly look if one file exist in the latest snapshot and also see the size. Here is how.

Determine latest snapshot

# restic snapshots | tail -10

09fdf7e5  2021-01-16 23:30:02  usph-vmli-do01  daily         
                                                             /tank01/backups/sites3

Find

# restic find -s 09fdf7e5 wpsites_dump.sql
repository 1144d629 opened successfully, password is correct
Found matching entries in snapshot 09fdf7e5 from 2021-01-16 23:30:02
/tank01/backups/sites3/databases/wpsites_dump.sql

List

# restic ls -l 09fdf7e5 /tank01/backups/sites3/databases/wpsites_dump.sql
repository 1144d629 opened successfully, password is correct
snapshot 09fdf7e5 of [/tank01/src /tank01/stack /tank01/backups/sites3] filtered by [/tank01/backups/sites3/databases/wpsites_dump.sql] at 2021-01-16 23:30:02.785360824 -0600 CST):
-rw-r--r--     0     0 1326541 2021-01-16 20:00:13 /tank01/backups/sites3/databases/wpsites_dump.sql

List Directory

root@desktop01:~# restic ls -l 09fdf7e5 /tank01/backups/sites3/databases
repository 1144d629 opened successfully, password is correct
snapshot 09fdf7e5 of [tank01/stack /tank01/backups/sites3] filtered by [/tank01/backups/sites3/databases] at 2021-01-16 23:30:02.785360824 -0600 CST):
drwxr-xr-x     0     0      0 2020-05-25 09:02:19 /tank01/backups/sites3/databases
...
-rw-r--r--     0     0 1326541 2021-01-16 20:00:13 /tank01/backups/sites3/databases/wpsites_dump.sql

Comments Off on Restic find and list one file
comments

Posted in restic

« Older Entries
Newer Entries »