Does anything other than datagrepper use the datanommer models API directly

[ryanlerch]

The datanommer readme states that the majority of tools and applications interact with the datanommer db using datagrepper, with the exception of datagrepper itself, and fedbadges.

The questions here are:

• does fedbadges still connect directly to the datanommer db?
• does anything else connect directly to the datanommer db?

[abompard]

Yeah, the badges do connect to the datanommer db directly using the models. We'll need to update this piece of code too.

[ryanlerch]

So did a little digging into this, and it appears that FMN also uses datanommer.models, but these are the only two i could find that use the database models directly. Both these applications seem to only use the grep method in datanommer.models, which is a good thing, as the API for this method has not changed, so we shouldnt need to change code in badges or fmn. We likely will need to update these applications to use the newest version of datanommer.models which shouldnt be a big issue.

Notes on where these two applications use datanommer.models are detailed below:

Fedora Badges

It appears that Fedora Badges is using the datanommer.models "grep" function in fedora-infra/fedbadges , which is the consumer that reads the message queue and awards the badges.

AFAICT, datanommer.models is initted here:

• https://github.com/fedora-infra/fedbadges/blob/develop/fedbadges/consumers.py#L125

and the grep method called in these two spots:

• https://github.com/fedora-infra/fedbadges/blob/develop/fedbadges/rules.py#L424
• https://github.com/fedora-infra/fedbadges/blob/develop/fedbadges/rules.py#L472

FMN

datanommer.models is initted here:

• https://github.com/fedora-infra/fmn/blob/develop/fmn/web/app.py#L77

and the grep method used once here:

• https://github.com/fedora-infra/fmn/blob/develop/fmn/web/app.py#L572

[ryanlerch]

OK,

Working on getting badges working in staging to see if we can test this.

[ryanlerch]

Ok, so badges is running in staging -- kinda.

The frontend is working -- but the backend fails awarding badges as it appears to stil interface with FAS2 (and there is no FAS2 for staging... so...)

Anyway, long story short, we can still test out the new datanommer by changing the url on staging badges-backend (/etc/fedmsg.d/datanommer.py) to be:

config = {
    'datanommer.sqlalchemy.url': 'postgresql://datanommer:ohZoad7Foowe2the4Zae5ovee8Iquaxi@db-datanommer01.stg.iad2.fedoraproject.org/datanommer2',
}

and we get this error:

Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: 'SELECT count(*) AS count_1 \nFROM (SELECT messages.id AS messages_id, messages.msg_id AS messages_msg_id, messages.i AS messages_i, messages.topic AS messages
_topic, messages.timestamp AS messages_timestamp, messages.certificate AS messages_certificate, messages.signature AS messages_signature, messages.category AS messages_category, messages.username AS messages_username, messages.crypto AS m
essages_crypto, messages.source_name AS messages_source_name, messages.source_version AS messages_source_version, messages._msg AS messages__msg, messages._headers AS messages__headers \nFROM messages \nWHERE (EXISTS (SELECT 1 \nFROM user_messages, "user" \nWHERE messages.id = user_messages.msg AND "user".name = user_messages.username AND "user".name = %(name_1)s)) AND messages.topic = %(topic_1)s) AS anon_1' {'name_1': u'ryanlerch', 'topic_1': u'org.fedoraproject.stg.wiki.article.edit'}                                                                                                                                                                                                                              Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: Traceback (most recent call last):                                                                                                                             Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib/python2.7/site-packages/fedbadges/utils.py", line 89, in inner                                                                                  
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return method(self, *args, **kwargs)                                                                                                                           Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 385, in matches                                                                               Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return self.specialization.matches(msg)                                                                                                                        Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 498, in matches                                                                               Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: total, pages, query = self._construct_query(msg)                                                                                                               Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib/python2.7/site-packages/fedbadges/rules.py", line 472, in _construct_query                                                                      Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: total, pages, query = datanommer.models.Message.grep(**kwargs)                                                                                                 Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib/python2.7/site-packages/datanommer/models/__init__.py", line 431, in grep                                                                       Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: total = query.count()                                                                                                                                          Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2640, in count                                                                         Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return self.from_self(col).scalar()                                                                                                                            Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2426, in scalar                                                                        Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: ret = self.one()                                                                                                                                               Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2395, in one                                                                           Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: ret = list(self)                                                                                                                                               Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2438, in __iter__                                                                      Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return self._execute_and_instances(context)                                                                                                                    Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances                                                        
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: result = conn.execute(querycontext.statement, self._params)                                                                                                    Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 729, in execute                                                                      Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return meth(self, multiparams, params)                                                                                                                         Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection                                                      Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: return connection._execute_clauseelement(self, multiparams, params)                                                                                            Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement                                                       Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: compiled_sql, distilled_params                                                                                                                                 Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context                                                             Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: context)                                                                                                                                                       Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1159, in _handle_dbapi_exception                                                     
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: exc_info                                                                                                                                                       
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause                                                             
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: reraise(type(exception), exception, tb=exc_tb)                                                                                                                 
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context                                                             
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: context)                                                                                                                                                       
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: cursor.execute(statement, parameters)                                                                                                                          
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: ProgrammingError: (ProgrammingError) column messages._msg does not exist                
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: LINE 2: ...ssages.source_version AS messages_source_version, messages._...                                                                                     
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: ^                                                                                                                                                              
Aug 23 02:54:58 badges-backend01.stg.iad2.fedoraproject.org fedmsg-hub[20127]: HINT:  Perhaps you meant to reference the column "messages.msg".

Which i guess is because we are running python2.7 on badges backend, and using python-datanommer-models 0.9.0 -- https://src.fedoraproject.org/rpms/python-datanommer-models (this is pulled in as a dep of python-fedbadges, which is the fedmsg backend consumer)

[ryanlerch]

However, this is where i stopped, as not sure what the best approach here is -- the two options i see are:

1. Somehow provide a python 2.7 version of datanommer.models -- not sure this is even possible with the timescaledb stuff we have added.
2. Update the entire badges stack to python3 (pretty sure given our time contraints, this is out of the picture too)

[abompard]

Another option would be to switch badges to use HTTP requests instead of using the models directly.

[mscherer]

Fedbadges (in the develop branch) was ported to python 3: fedora-infra/fedbadges@a3f2b03

However, I think no one really know how to do a release (especially since the CI was not working), and the plan was to just run it in a pod, with the HEAD version (since doing release for a custom software where we are the only user seems a bit overkill)

I am not sure there is no need to port the whole stack, just fedbadges.

[ryanlerch]

@mscherer fedbadges / badges backend on fedora infra is currently in a VM (i.e. not in openshift) AFAICT

should the badges team open a ticket to get a new VM set up for fedbadges like what was done in this ticket for FMN:

https://pagure.io/fedora-infrastructure/issue/9746

from the datanommer side, once fedbadges is updated to python3, fedbages can use the newest version of datanommer.models, and that side of things should just work as it has in the past (the datanommer.models API fedbadges uses hasnt changed)

[ryanlerch]

okay, opened a infra ticket for this -- not really sure how to proceed though.

https://pagure.io/fedora-infrastructure/issue/10192