Uipa containerization #105

sthapa · 2024-08-14T00:12:20Z

This PR has the changes and files needed to run the uipa website using docker/podman compose. You'll need to build the uipa image using docker build -f Dockerfile . -t uipa_backend:0.10 --platform=linux/amd64 and then running docker compose -f docker-compose.local.yml up should bring everything up. The uipa website should be available at http://127.0.0.1:8000 after a few minutes. The uipa container uses alpine and runs the website using django 4.2.15 and python 3.12.

sthapa · 2024-08-14T00:13:10Z

I've tested this on OS X and fedora 40 so it should be cross platform compatible.

kobebuckley

Tested on my local VSCode and default Github codespaces ( which uses server http://0.0.0.0:8000/ ). Both worked great for me.

sthapa · 2024-08-14T06:16:27Z

Tested on my local VSCode and default Github codespaces ( which uses server http://0.0.0.0:8000/ ). Both worked great for me.

@kobebuckley , can this replace PR #46 or do we need to do more to get codespaces working?

kobebuckley · 2024-08-14T14:07:45Z

Tested on my local VSCode and default Github codespaces ( which uses server http://0.0.0.0:8000/ ). Both worked great for me.

@kobebuckley , can this replace PR #46 or do we need to do more to get codespaces working?

@sthapa Yes, this can replace PR #46.

russtoku

Works well up to bringing up the front page of the dev website but clicking on links results in errors:

TemplateDoesNotExist
- Requests
- Public Agencies
- Login
- Search for requests
- Search for information
relation "django_redirect" does not exist
- About (top and bottom of front page)
- Help (top of front page)
- FAQ (bottom of front page)
- Terms of Use (bottom of front page)
- Privacy Statement (bottom of front page)

russtoku · 2024-08-15T20:32:11Z

A. TemplateDoesNotExist

From the logs when clicking on the Requests link:

app-1  | Internal Server Error: /requests/
app-1  | Traceback (most recent call last):
app-1  |   File "/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
app-1  |     response = get_response(request)
app-1  |                ^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/django/core/handlers/base.py", line 220, in _get_response
app-1  |     response = response.render()
app-1  |                ^^^^^^^^^^^^^^^^^
app-1  |   File "/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/django/template/response.py", line 114, in render
app-1  |     self.content = self.rendered_content
app-1  |                    ^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/django/template/response.py", line 90, in rendered_content
app-1  |     template = self.resolve_template(self.template_name)
app-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/django/template/response.py", line 72, in resolve_template
app-1  |     return select_template(template, using=self.using)
app-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
app-1  |   File "/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/django/template/loader.py", line 47, in select_template
app-1  |     raise TemplateDoesNotExist(", ".join(template_name_list), chain=chain)
app-1  | django.template.exceptions.TemplateDoesNotExist: foirequest/list.html, foirequest/foirequest_list.html

The foirequest/list.html template file is supposed be in the foirequest/templates/ directory under the froide package in site-packages. It's not there. It might be better to have the froide package be installed as an editable package under the same directory that uipa is installed in (/srvr/django/).

When I check in the app container, I see:

~/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages # ls froide
__init__.py         celery.py           follow              problem             tests
__pycache__         comments            frontpage           proof               upload
accesstoken         conftest.py         georegion           publicbody          urls.py
account             document            guide               routing.py          wsgi.py
api.py              foirequest          helper              settings.py
bounce              foirequestfollower  letter              team
campaign            foisite             organization        templates

~/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages # cd froide

~/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/froide # ls templates/
400.html             admin                footer.html          index.html           snippets
403.html             admin_utils          header.html          mfa
404.html             base.html            header_minimal.html  pagination
500.html             emails               header_reduced.html  scaffold.html
_frontend.html       error_base.html      help                 simple_base.html

~/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/froide # find . -name list.html
~/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/froide #

B. STATICFILES_DIRS setting does not exist.

app-1  | django-configurations version 2.5.1, using configuration Dev
app-1  | Performing system checks...
app-1  | 
app-1  | /root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/weasyprint/text/fonts.py:65: UserWarning: No fonts configured in FontConfig. Expect ugly output.
app-1  |   warn('No fonts configured in FontConfig. Expect ugly output.')
app-1  | System check identified some issues:
app-1  | 
app-1  | WARNINGS:
app-1  | ?: (staticfiles.W004) The directory '/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/froide/static' in the STATICFILES_DIRS setting does not exist.
app-1  | 
app-1  | System check identified 1 issue (0 silenced).
app-1  | 
app-1  | You have 2 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): redirects.
app-1  | Run 'python manage.py migrate' to apply them.
app-1  | August 15, 2024 - 22:15:43
app-1  | Django version 4.2.15, using settings 'uipa_org.settings.development'
app-1  | Starting development server at http://0.0.0.0:8000/
app-1  | Quit the server with CONTROL-C.

The dev settings needs to be fixed.

russtoku · 2024-08-16T06:28:26Z

Also, the elasticsearch server URL isn't correct as show by this error:

app-1            |   File "/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/elastic_transport/_transport.py", line 342, in perform_request                                                                   
app-1            |     resp = node.perform_request(                                                                     
app-1            |            ^^^^^^^^^^^^^^^^^^^^^                                                                     
app-1            |   File "/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/elastic_transport/_node/_http_urllib3.py", line 202, in perform_request                                                          
app-1            |     raise err from None                                                                              
app-1            | elastic_transport.ConnectionError: Connection error caused by: ConnectionError(Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe118c7d190>: Failed to establish a new connection: [Errno 111] Connection refused))

I think that this needs to be changed in the Dev settings:

ELASTICSEARCH_DSL  = {'default': {'hosts': 'http://localhost:9200', 'http_auth': ('elastic', 'froide')}}

The localhost:9200 should probably be elasticsearch:9200 because the container started by docker compose up is on a different host from the Django dev server.

sthapa · 2024-08-19T04:46:41Z

A. TemplateDoesNotExist
...
When I check in the app container, I see:

Yeah, the froide code doesn't get packaged (e.g. python setup.py sdist) correctly in alpine and debian and possibly elsewhere. I'm looking into seeing why that is.

B. STATICFILES_DIRS setting does not exist.

app-1  | System check identified some issues:
app-1  | 
app-1  | WARNINGS:
app-1  | ?: (staticfiles.W004) The directory '/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/froide/static' in the STATICFILES_DIRS setting does not exist.
app-1  | 
app-1  | System check identified 1 issue (0 silenced).
app-1  | 

The dev settings needs to be fixed.

I don't think that's a important concern for now. The other STATICFILES_DIRS that django looks for are present.

sthapa · 2024-08-19T04:49:51Z

Also, the elasticsearch server URL isn't correct as show by this error:

app-1            |   File "/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/elastic_transport/_transport.py", line 342, in perform_request                                                                   
app-1            |     resp = node.perform_request(                                                                     
app-1            |            ^^^^^^^^^^^^^^^^^^^^^                                                                     
app-1            |   File "/root/.cache/pypoetry/virtualenvs/uipa-JzPo6xSy-py3.12/lib/python3.12/site-packages/elastic_transport/_node/_http_urllib3.py", line 202, in perform_request                                                          
app-1            |     raise err from None                                                                              
app-1            | elastic_transport.ConnectionError: Connection error caused by: ConnectionError(Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe118c7d190>: Failed to establish a new connection: [Errno 111] Connection refused))

I think that this needs to be changed in the Dev settings:

ELASTICSEARCH_DSL  = {'default': {'hosts': 'http://localhost:9200', 'http_auth': ('elastic', 'froide')}}

The localhost:9200 should probably be elasticsearch:9200 because the container started by docker compose up is on a different host from the Django dev server.

I'd prefer to change that through an env injection in the docker compose rather than modifying the config files within the container. I believe there's a bug that prevents the code from picking it up from the env though so that'll need to be fixed.

sthapa · 2024-08-20T23:13:19Z

@russtoku the template and ES issues should be fixed now

russtoku

While the dev website does come up, it doesn't have an admin user and seed data (FOI laws, juridictions, public bodies, public body tags, and flatpages). If this state is OK, then we're good to go. However, I think that it's not helpful enough to encourage new contributors. I think that we can do better.

See the How the database is seeded docs.

BTS, an admin user can be created by loading the data from a fixture instead of interactively with "python manage.py createsuperuser".

run-backend.sh

setup-uipa.sh

uipa_org/settings/development.py

russtoku · 2024-08-30T04:07:13Z

OK with me to merge but I didn't test the set up in a GitHub Codespace.

sthapa and others added 2 commits August 13, 2024 10:45

Add files for containerization

11274e3

Remove requirements.txt

2ffb951

sthapa requested a review from a team as a code owner August 14, 2024 00:12

sthapa requested review from russtoku and tyliec August 14, 2024 00:13

kobebuckley reviewed Aug 14, 2024

View reviewed changes

russtoku reviewed Aug 15, 2024

View reviewed changes

sthapa and others added 4 commits August 19, 2024 17:35

More fixes for containerization

76f7e8c

Remove templates tarball, froide code fixed

f58395d

Update poetry lock file

d329acc

Make ES auth configurable, additional container fixes

76ab667

russtoku reviewed Aug 23, 2024

View reviewed changes

run-backend.sh Show resolved Hide resolved

run-backend.sh Outdated Show resolved Hide resolved

setup-uipa.sh Outdated Show resolved Hide resolved

uipa_org/settings/development.py Show resolved Hide resolved

sthapa and others added 2 commits August 24, 2024 09:03

Fix typo in name

a51a87e

Set gdal and geos lib locations in settings

9440c0b

russtoku approved these changes Aug 30, 2024

View reviewed changes

sthapa merged commit 2e88a29 into main Aug 30, 2024

sthapa deleted the containerization branch August 30, 2024 21:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uipa containerization #105

Uipa containerization #105

sthapa commented Aug 14, 2024 •

edited

Loading

sthapa commented Aug 14, 2024

kobebuckley left a comment

sthapa commented Aug 14, 2024

kobebuckley commented Aug 14, 2024

russtoku left a comment •

edited

Loading

russtoku commented Aug 15, 2024 •

edited

Loading

russtoku commented Aug 16, 2024 •

edited

Loading

sthapa commented Aug 19, 2024

sthapa commented Aug 19, 2024

sthapa commented Aug 20, 2024

russtoku left a comment

russtoku commented Aug 30, 2024

Uipa containerization #105

Uipa containerization #105

Conversation

sthapa commented Aug 14, 2024 • edited Loading

sthapa commented Aug 14, 2024

kobebuckley left a comment

Choose a reason for hiding this comment

sthapa commented Aug 14, 2024

kobebuckley commented Aug 14, 2024

russtoku left a comment • edited Loading

Choose a reason for hiding this comment

russtoku commented Aug 15, 2024 • edited Loading

russtoku commented Aug 16, 2024 • edited Loading

sthapa commented Aug 19, 2024

sthapa commented Aug 19, 2024

sthapa commented Aug 20, 2024

russtoku left a comment

Choose a reason for hiding this comment

russtoku commented Aug 30, 2024

sthapa commented Aug 14, 2024 •

edited

Loading

russtoku left a comment •

edited

Loading

russtoku commented Aug 15, 2024 •

edited

Loading

russtoku commented Aug 16, 2024 •

edited

Loading