tilt-hub is a document storage for transparency information specified in the Transparency Information Language and is part of the Transparency Information Language and Toolkit.
This document storage comprises five services in order to perform CRUD operations on transparency information documents and serves for clients, that either only want plain information about GDPR compliance of a data controller or carry out arbitrary complex research/statistics/analysis tasks.
- Create a MongoDB keyfile via
# Create secret
openssl rand -base64 756 > mongodb.keyfile
# Protect file
chmod 400 mongodb.keyfile
# Set owner
sudo chown systemd-coredump mongodb.keyfile
# Check if permissions look like this
ls -la mongodb.keyfile
-r-------- 1 systemd-coredump docker 1024 Feb 22 12:11 mongodb.keyfile- Run (
-dfor running in the background)
docker-compose up [-d]- After some startup time, there will be the following five containers. You're done!
docker-compose ps
Name Command State Ports
---
mongo docker-entrypoint.sh --con ... Up 0.0.0.0:27017->27017/tcp, 0.0.0.0:27018->27018/tcp, 0.0.0.0:27019->27019/tcp
mongo-express tini -- /docker-entrypoint ... Up 0.0.0.0:8081->8081/tcp
mongoke dumb-init -- /entrypoint.s ... Up 0.0.0.0:8082->80/tcp
restheart java -Dfile.encoding=UTF-8 ... Up 4443/tcp, 8009/tcp, 0.0.0.0:8080->8080/tcp
triggers docker-entrypoint.sh node ... Up(Optionally, in case of trouble)
- Stop and delete all running containers
docker compose downanddocker rm $(docker ps -aq)(attention, this deletes all containers!). - Check if there is an empty
volumes/mongodirectory. - Again, create a fresh keyfile (see above).
- Run only the mongo container via
docker compose up mongo --force-recreate. - Follow the steps below to enable the MongoDB replica set.
- Start all remaining containers
docker compose up.
Follow these steps to enable the MongoDB replica set
# Login into the container
docker exec -it mongo bin/bash
# Login to mongo shell
mongo
# Change current database
use admin
# Authenticate to the admin database
db.auth("root", "SuperSecret")
# Initiate replica set with one member (localhost)
rs.initiate({_id:"rs0", members: [{"_id": 0, "host": "localhost:27017"}]})
# Close mongo shell
exit
# Exit container shell
exitThe mongo database is available via mongodb://root:SuperSecret@mongo. A client application written in Python could look like:
from pymongo import MongoClient
from pprint import pprint
client = MongoClient('mongodb://root:SuperSecret@mongo:27017/?authSource=admin&readPreference=primary')
filter={
'controller.name': 'Green Company'
}
result = client['tilt']['tilt'].find(
filter=filter
)
for document in result:
pprint(document)A graphical interface to the database can be reached under http://localhost:8081/.
Username: admin
Password: SuperSecret
In order to retrieve all documents from the database tilt and collection tilt.
http GET "localhost:8080/tilt/tilt" -a admin:secret
HTTP/1.1 200 OK
Content-Type: application/json
Date: Sun, 16 Aug 2020 21:36:05 GMT
...
[
{
"_etag": {
"$oid": "5f39a2cb5937e6719cbb3bed"
},
"_id": {
"$oid": "5f39a2cb8f9a233727fb7fa2"
},
"accessAndDataPortability": {
"administrativeFee": {
"amount": 0,
"currency": "EUR"
},
...Or retrieve a specific document using its id:
http GET "http://localhost:8080/tilt/tilt/6053f42d0da4272ad0d52e8a" -a admin:secretNew documents can be created by POSTing a valid TILT document to http://localhost:8080/tilt/tilt (don't forget to authorize with Basic Auth).
See here for all query possibilites.
Some examples for GETing documents:
GET "http://localhost:8080/tilt/tilt/605bc9b783990f850a9cf492"
single tilt document by id 605…
GET "http://localhost:8080/tilt/tilt"
all tilt documents
GET "http://localhost:8080/tilt/tilt?filter={'meta.name' : 'Blue'}"
all tilt documents where meta.name equals 'Blue'
GET "http://localhost:8080/tilt/tilt?filter={'meta.status': 'inactive', 'dataDisclosed.recipients.name': 'Red'}"
all controllers that send data to the Red company and whose document status is inactive
GET "http://localhost:8080/tilt/tilt?keys={'controller.name' : 1}"
all tilt documents but only the controller.name attribute (0 for all but controller.name)You can use filter to have all MongoDB’s queries power.
Don't forget to authorize using Basic Auth (User: admin, Password: secret).
Use the GraphiQL interface running at http://localhost:8082 to perform GraphQL operations. To return the meta.name field of all documents run:
{
TiltNodes(first: 1000) {
edges {
node {
meta {
name
}
}
}
}
}In Python, you could retrieve the same information like this:
from graphqlclient import GraphQLClient
client = GraphQLClient('http://tilt-hub-host.com:8082/')
result = client.execute('''
query {
TiltNodes {
edges {
node {
meta {
name
}
}
}
}
}
''')
print(result)In this basic scenario there will be a JavaScript trigger microservice which validates all documents on every change that occurs using the tilt-schema.
The output of the validation can be obtained directly from the logs:
docker-compose logs -f triggers
triggers | Downloaded schema from https://raw.githubusercontent.com/Transparency-Information-Language/schema/master/tilt-schema.json
triggers | Trigger on tilt collection initialized successfully!
triggers | Document changes detected!
triggers | tilt schema validation successful!Similarly, there could by a Python based trigger which is located here. In order to use it, configure the docker-compose.yml accordingly.
Elias Grünewald