From 3693f24f9ffc338a031ff2be102afdf0fca5a42e Mon Sep 17 00:00:00 2001 From: Andrew Pontzen Date: Fri, 10 Nov 2023 09:19:05 +0000 Subject: [PATCH 1/2] Update database info including integrating #218 --- docs/rdbms.md | 49 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 46 insertions(+), 3 deletions(-) diff --git a/docs/rdbms.md b/docs/rdbms.md index 87f914d8..198d1754 100644 --- a/docs/rdbms.md +++ b/docs/rdbms.md @@ -10,7 +10,14 @@ SQLite SQLite is the default database. It is simple in the sense that it keeps your entire database within a single file which can easily be transferred to different systems. Additionally, the SQLite driver is included with Python and so it's quick to get started. -There are two major, related drawbacks to SQLite. The first is that the +There are two major, related drawbacks to SQLite. The first is that to access it one must copy over +the file, and there is no automated way to keep files synchronised between hosts. (Probably the best +thing to do is to write to the database only on one cluster, and then `rsync` it to the relevant +analysis machines.) The second is that it is not really designed for parallel writes, so when tangos +is writing to the database it must manually try to synchronise writes between different workers. +Tangos does a pretty good job of this, but some network file systems can be slow at releasing file +locks that SQLite uses extensively. If you run into errors about 'database is locked', you have reached +the limit of how many tangos processes can safely write to SQLite simultaneously. PostgreSQL and MySQL -------------------- @@ -56,5 +63,41 @@ or for PostgreSQL: export TANGOS_DB_CONNECTION=postgresql+psycopg2://tangos:my_secret_password@localhost/database_name ``` -You can now use all the tangos tools as normal, and they will populate the MySQL/PostgreSQL database -instead of a SQLite file. + + You can now create new users that can access your mysql server with their own username and password. + + ```bash + echo "create user 'my_new_user'@'%' identified by 'new_password';" | docker exec -i mysql-server mysql -pmy_secret_password + ``` + + Note that in MySQL the `%` acts as a wild card, so this command creates a new user + logging in from any host. + + The new user would then connect to the database: + + ```bash + export TANGOS_DB_CONNECTION=mysql+pymysql://my_new_user:new_password@localhost:3306/database_name + ``` + +The database can be accessed remotely if allowed by any applicable firewalls, by replacing `localhost` +with the actual host like `fancy_computer.astro.fancy_school.edu`. Note, however, that +running a database server open to the world has security implications and may be disallowed by +relevant institutions. The simplest approach, rather than opening up firewalls, is to tunnel in. +For example, the server can be accessed as though it's running on `localhost` if the user +first ssh tunnels into `fancy_computer.astro.fancy_school.edu`: + + ```bash + ssh -N -f -L localhost:3306:localhost:3306 my_username@fancy_computer.astro.fancy_school.edu + ``` + +Note that new users will by default only be able to view a database. Granting +additional permissions should be done on a case-by-case basis. Only the root user can +do this by defualt. To give a user complete permission to edit an existing database: + + ```bash + echo "grant all on database_name.* to 'new_user'@'%';" | docker exec -i mysql-server mysql -pmy_secret_password + echo "flush privileges;" | docker exec -i mysql-server mysql -pmy_secret_password + ``` + + You (and whatever users you choose) can now use all the tangos tools as normal, and they will + populate the MySQL/PostgreSQL database instead of a SQLite file. From 75de302b4418e1e648b29e6ff0e13f19fea56b89 Mon Sep 17 00:00:00 2001 From: Andrew Pontzen Date: Fri, 10 Nov 2023 09:21:43 +0000 Subject: [PATCH 2/2] Fix pre-commit test --- docs/rdbms.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/rdbms.md b/docs/rdbms.md index 198d1754..3ff63496 100644 --- a/docs/rdbms.md +++ b/docs/rdbms.md @@ -14,10 +14,10 @@ There are two major, related drawbacks to SQLite. The first is that to access it the file, and there is no automated way to keep files synchronised between hosts. (Probably the best thing to do is to write to the database only on one cluster, and then `rsync` it to the relevant analysis machines.) The second is that it is not really designed for parallel writes, so when tangos -is writing to the database it must manually try to synchronise writes between different workers. +is writing to the database it must manually try to synchronise writes between different workers. Tangos does a pretty good job of this, but some network file systems can be slow at releasing file locks that SQLite uses extensively. If you run into errors about 'database is locked', you have reached -the limit of how many tangos processes can safely write to SQLite simultaneously. +the limit of how many tangos processes can safely write to SQLite simultaneously. PostgreSQL and MySQL -------------------- @@ -67,10 +67,10 @@ export TANGOS_DB_CONNECTION=postgresql+psycopg2://tangos:my_secret_password@loca You can now create new users that can access your mysql server with their own username and password. ```bash - echo "create user 'my_new_user'@'%' identified by 'new_password';" | docker exec -i mysql-server mysql -pmy_secret_password + echo "create user 'my_new_user'@'%' identified by 'new_password';" | docker exec -i mysql-server mysql -pmy_secret_password ``` - Note that in MySQL the `%` acts as a wild card, so this command creates a new user + Note that in MySQL the `%` acts as a wild card, so this command creates a new user logging in from any host. The new user would then connect to the database: @@ -82,7 +82,7 @@ export TANGOS_DB_CONNECTION=postgresql+psycopg2://tangos:my_secret_password@loca The database can be accessed remotely if allowed by any applicable firewalls, by replacing `localhost` with the actual host like `fancy_computer.astro.fancy_school.edu`. Note, however, that running a database server open to the world has security implications and may be disallowed by -relevant institutions. The simplest approach, rather than opening up firewalls, is to tunnel in. +relevant institutions. The simplest approach, rather than opening up firewalls, is to tunnel in. For example, the server can be accessed as though it's running on `localhost` if the user first ssh tunnels into `fancy_computer.astro.fancy_school.edu`: @@ -95,9 +95,9 @@ additional permissions should be done on a case-by-case basis. Only the root use do this by defualt. To give a user complete permission to edit an existing database: ```bash - echo "grant all on database_name.* to 'new_user'@'%';" | docker exec -i mysql-server mysql -pmy_secret_password - echo "flush privileges;" | docker exec -i mysql-server mysql -pmy_secret_password + echo "grant all on database_name.* to 'new_user'@'%';" | docker exec -i mysql-server mysql -pmy_secret_password + echo "flush privileges;" | docker exec -i mysql-server mysql -pmy_secret_password ``` - You (and whatever users you choose) can now use all the tangos tools as normal, and they will + You (and whatever users you choose) can now use all the tangos tools as normal, and they will populate the MySQL/PostgreSQL database instead of a SQLite file.