Terraform Stateserver Using Python
Terraform can utilize a http backend for maintaining state. This is a test of a Terraform http backend using a server implemented with python.
NOTE: checked source into https://github.com/rrossouw01/terraform-stateserver-py/
recipe and components
Using Virtualbox Ubuntu 20.10 and followed links:
- https://www.madebymikal.com/writing-a-terraform-remote-state-server/
- https://github.com/mikalstill/junkcode/tree/master/terraform/remote_state
setup
$ mkdir tf-state-server
$ cd tf-state-server
$ virtualenv -p python3 venv
created virtual environment CPython3.8.6.final.0-64 in 174ms
creator CPython3Posix(dest=/home/rrosso/tf-state-server/venv, clear=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/rrosso/.local/share/virtualenv)
added seed packages: pip==20.1.1, pkg_resources==0.0.0, setuptools==44.0.0, wheel==0.34.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
$ source venv/bin/activate
(venv) $ pip install -U -r requirements.txt
Collecting flask
Using cached Flask-1.1.2-py2.py3-none-any.whl (94 kB)
Collecting flask_restful
Downloading Flask_RESTful-0.3.8-py2.py3-none-any.whl (25 kB)
Collecting itsdangerous>=0.24
Using cached itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting Werkzeug>=0.15
Using cached Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
Collecting Jinja2>=2.10.1
Using cached Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
Collecting click>=5.1
Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Collecting pytz
Downloading pytz-2021.1-py2.py3-none-any.whl (510 kB)
|████████████████████████████████| 510 kB 3.0 MB/s
Collecting six>=1.3.0
Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting aniso8601>=0.82
Downloading aniso8601-9.0.1-py2.py3-none-any.whl (52 kB)
|████████████████████████████████| 52 kB 524 kB/s
Collecting MarkupSafe>=0.23
Using cached MarkupSafe-1.1.1-cp38-cp38-manylinux2010_x86_64.whl (32 kB)
Installing collected packages: itsdangerous, Werkzeug, MarkupSafe, Jinja2, click, flask, pytz, six, aniso8601, flask-restful
Successfully installed Jinja2-2.11.3 MarkupSafe-1.1.1 Werkzeug-1.0.1 aniso8601-9.0.1 click-7.1.2 flask-1.1.2 flask-restful-0.3.8 itsdangerous-1.1.0 pytz-2021.1 six-1.15.0
(venv) $ python3 stateserver.py
* Serving Flask app "stateserver" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://192.168.1.235:5000/ (Press CTRL+C to quit)
...terraform point to remote http
➜ cat main.tf
terraform {
backend "http" {
address = "http://192.168.1.235:5000/terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a"
lock_address = "http://192.168.1.235:5000/terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a"
lock_method = "PUT"
unlock_address = "http://192.168.1.235:5000/terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a"
unlock_method = "DELETE"
}
}
➜ source ../env-vars
➜ terraform init
Initializing the backend...
Do you want to copy existing state to the new backend?
Pre-existing state was found while migrating the previous "local" backend to the
newly configured "http" backend. No existing state was found in the newly
configured "http" backend. Do you want to copy this state to the new "http"
backend? Enter "yes" to copy and "no" to start with an empty state.
Enter a value: yes
Successfully configured the backend "http"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.
* provider.oci: version = "~> 4.17"
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
server shows
---
````bash
...
192.168.1.111 - - [16/Mar/2021 10:50:00] "POST /terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a?ID=84916e49-1b44-1b32-2058-62f28e1e8ee7 HTTP/1.1" 200 -
192.168.1.111 - - [16/Mar/2021 10:50:00] "DELETE /terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 -
192.168.1.111 - - [16/Mar/2021 10:50:00] "GET /terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 -
...
$ ls -la .stateserver/
total 24
drwxrwxr-x 2 rrosso rrosso 4096 Mar 16 10:50 .
drwxrwxr-x 4 rrosso rrosso 4096 Mar 16 10:43 ..
-rw-rw-r-- 1 rrosso rrosso 4407 Mar 16 10:50 4cdd0c76-d78b-11e9-9bea-db9cd8374f3a
-rw-rw-r-- 1 rrosso rrosso 5420 Mar 16 10:50 4cdd0c76-d78b-11e9-9bea-db9cd8374f3a.log
$ more .stateserver/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a*
::::::::::::::
.stateserver/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a
::::::::::::::
{
"lineage": "9b756fb7-e41a-7cd6-d195-d794f377e7be",
"outputs": {},
"resources": [
{
"instances": [
{
"attributes": {
"compartment_id": null,
"id": "ObjectStorageNamespaceDataSource-0",
"namespace": "axwscg6apasa"
},
"schema_version": 0
}
],
...
"serial": 0,
"terraform_version": "0.12.28",
"version": 4
}
::::::::::::::
.stateserver/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a.log
::::::::::::::
lock: {
"Created": "2021-03-16T15:49:34.567178267Z",
"ID": "7eea0d70-5f53-e475-041b-bcc393f4a92d",
"Info": "",
"Operation": "migration destination state",
"Path": "",
"Version": "0.12.28",
"Who": "rrosso@desktop01"
}
unlock: {
"Created": "2021-03-16T15:49:34.567178267Z",
"ID": "7eea0d70-5f53-e475-041b-bcc393f4a92d",
"Info": "",
"Operation": "migration destination state",
"Path": "",
"Version": "0.12.28",
"Who": "rrosso@desktop01"
}
lock: {
"Created": "2021-03-16T15:49:45.760917508Z",
"ID": "84916e49-1b44-1b32-2058-62f28e1e8ee7",
"Info": "",
"Operation": "migration destination state",
"Path": "",
"Version": "0.12.28",
"Who": "rrosso@desktop01"
}
state_write: {
"lineage": "9b756fb7-e41a-7cd6-d195-d794f377e7be",
"outputs": {},
"resources": [
{
"instances": [
{
"attributes": {
"compartment_id": null,
"id": "ObjectStorageNamespaceDataSource-0",
"namespace": "axwscg6apasa"
},
"schema_version": 0
}
...stateserver shows during plan
...
192.168.1.111 - - [16/Mar/2021 10:54:12] "PUT /terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 -
192.168.1.111 - - [16/Mar/2021 10:54:12] "GET /terraform_state/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 -
192.168.1.111 - - [16/Mar/2021 10:54:15] "DELETE /terraform_lock/4cdd0c76-d78b-11e9-9bea-db9cd8374f3a HTTP/1.1" 200 -
...source
$ more requirements.txt stateserver.py
::::::::::::::
requirements.txt
::::::::::::::
flask
flask_restful
::::::::::::::
stateserver.py
::::::::::::::
#!/usr/bin/python3
import flask
import flask_restful
import json
import logging
import os
app = flask.Flask(__name__)
api = flask_restful.Api(app)
@app.before_request
def log_request_info():
headers = []
for header in flask.request.headers:
headers.append('%s = %s' % (header[0], header[1]))
body = flask.request.get_data().decode('utf-8').split('\n')
app.logger.debug(('%(method)s for %(url)s...\n'
' Header -- %(headers)s\n'
' Body -- %(body)s\n')
% {
'method': flask.request.method,
'url': flask.request.url,
'headers': '\n Header -- '.join(headers),
'body': '\n '.join(body)
})
class Root(flask_restful.Resource):
def get(self):
resp = flask.Response(
'Oh, hello',
mimetype='text/html')
resp.status_code = 200
return resp
class StateStore(object):
def __init__(self, path):
self.path = path
os.makedirs(self.path, exist_ok=True)
def _log(self, id, op, data):
log_file = os.path.join(self.path, id) + '.log'
with open(log_file, 'a') as f:
f.write('%s: %s\n' %(op, data))
def get(self, id):
file = os.path.join(self.path, id)
if os.path.exists(file):
with open(file) as f:
d = f.read()
self._log(id, 'state_read', {})
return json.loads(d)
return None
def put(self, id, info):
file = os.path.join(self.path, id)
data = json.dumps(info, indent=4, sort_keys=True)
with open(file, 'w') as f:
f.write(data)
self._log(id, 'state_write', data)
def lock(self, id, info):
# NOTE(mikal): this is racy, but just a demo
lock_file = os.path.join(self.path, id) + '.lock'
if os.path.exists(lock_file):
# If the lock exists, it should be a JSON dump of information about
# the lock holder
with open(lock_file) as f:
l = json.loads(f.read())
return False, l
data = json.dumps(info, indent=4, sort_keys=True)
with open(lock_file, 'w') as f:
f.write(data)
self._log(id, 'lock', data)
return True, {}
def unlock(self, id, info):
lock_file = os.path.join(self.path, id) + '.lock'
if os.path.exists(lock_file):
os.unlink(lock_file)
self._log(id, 'unlock', json.dumps(info, indent=4, sort_keys=True))
return True
return False
state = StateStore('.stateserver')
class TerraformState(flask_restful.Resource):
def get(self, tf_id):
s = state.get(tf_id)
if not s:
flask.abort(404)
return s
def post(self, tf_id):
print(flask.request.form)
s = state.put(tf_id, flask.request.json)
return {}
class TerraformLock(flask_restful.Resource):
def put(self, tf_id):
success, info = state.lock(tf_id, flask.request.json)
if not success:
flask.abort(423, info)
return info
def delete(self, tf_id):
if not state.unlock(tf_id, flask.request.json):
flask.abort(404)
return {}
api.add_resource(Root, '/')
api.add_resource(TerraformState, '/terraform_state/')
api.add_resource(TerraformLock, '/terraform_lock/')
if __name__ == '__main__':
# Note this is not run with the flask task runner...
app.log = logging.getLogger('werkzeug')
app.log.setLevel(logging.DEBUG)
#app.run(host='0.0.0.0', debug=True) 