Feature/3959 bg monitoring fixed #2423

richard-jones · 2024-10-11T10:19:25Z

Issue: https://github.com/DOAJ/doajPM/issues/3959

Replaces PR: #2419

update bg monitoring settings for mission critical monitoring changes

Applies tighter monitoring constraints on the background jobs, as per this spreadsheet: https://docs.google.com/spreadsheets/d/1XWq-7R9hZmZsBA27eOgs9cO-SfJnLEE0dW50DVtKhpI/edit?gid=543591434#gid=543591434

This PR...

Developer Checklist

Developers should review and confirm each of these items before requesting review

Code meets acceptance criteria from issue
Unit tests are written and all pass
User Test Scripts (if required) are written and have been run through
Project's coding standards are met
- No deprecated methods are used
- No magic strings/numbers - all strings are in constants or messages files
- ES queries are wrapped in a Query object rather than inlined in the code
- Where possible our common library functions have been used (e.g. dates manipulated via dates)
- Cleaned up commented out code, etc
- Urls are constructed with url_for not hard-coded
Code documentation and related non-code documentation has all been updated
- Core model documentation has been added to if needed: https://docs.google.com/spreadsheets/d/1lun2S9vwGbyfy3WjIjgXBm05D-3wWDZ4bp8xiIYfImM/edit
- Events and consumers documentation has been added if needed: https://docs.google.com/spreadsheets/d/1oIeG5vg-blm2MZCE-7YhwulUlSz6TOUeY8jAftdP9JE/edit
Migation has been created and tested
There is a recent merge from develop

Reviewer Checklist

Reviewers should review and confirm each of these items before approval
If there are multiple reviewers, this section should be duplicated for each reviewer

Code meets acceptance criteria from issue
Unit tests are written and all pass
User Test Scripts (if required) are written and have been run through
Project's coding standards are met
- No deprecated methods are used
- No magic strings/numbers - all strings are in constants or messages files
- ES queries are wrapped in a Query object rather than inlined in the code
- Where possible our common library functions have been used (e.g. dates manipulated via dates)
- Cleaned up commented out code, etc
- Urls are constructed with url_for not hard-coded
Code documentation and related non-code documentation has all been updated
- Core model documentation has been added to if needed: https://docs.google.com/spreadsheets/d/1lun2S9vwGbyfy3WjIjgXBm05D-3wWDZ4bp8xiIYfImM/edit
- Events and consumers documentation has been added if needed: https://docs.google.com/spreadsheets/d/1oIeG5vg-blm2MZCE-7YhwulUlSz6TOUeY8jAftdP9JE/edit
Migation has been created and tested
There is a recent merge from develop

Testing

This tightens the monitoring on a number of bg jobs, so would be good to validate that this won't cause any major surge in unnecessary reporting.

Probably need to send it live and then adjust.

Deployment

Configuration changes

Adds new values in:

BG_MONITOR_QUEUED_CONFIG
BG_MONITOR_ERRORS_CONFIG
BG_MONITOR_DEFAULT_CONFIG

Monitoring

This may trigger new types of "unstable" alerts in the background jobs reporting, so we should ensure that those alerts come through

… also resolve merge conflict

RK206

Test cases failing from local. Can you please check. Are there any configuration changes required to test?

RK206 · 2024-11-04T05:02:28Z

doajtest/unit/test_background_task_status.py

Failed tests on my local. Not sure if these are specific to my local. Fixed import issue on my local in portality/bll/services/background_task_status.py before testing.

doajtest/unit/test_background_task_status.py:178 (TestBackgroundTaskStatus.test_create_background_status__error_in_period_not_found) self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__error_in_period_not_found> @apply_test_case_config(bg_monitor_errors_config__a) def test_create_background_status__error_in_period_not_found(self): save_mock_bgjob(JournalCSVBackgroundTask.__action__, status=constants.BGJOB_STATUS_ERROR, created_before_sec=1000000000) status_dict = background_task_status.create_background_status() journal_csv_dict = status_dict['queues']['scheduled_short']['errors'].get(JournalCSVBackgroundTask.__action__, {}) > assert is_stable(status_dict['status']) E AssertionError: assert False E + where False = is_stable('unstable') test_background_task_status.py:189: AssertionError

doajtest/unit/test_background_task_status.py:106 (TestBackgroundTaskStatus.test_create_background_status__invalid_last_completed__events_queue) self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__invalid_last_completed__events_queue> @apply_test_case_config(bg_monitor_last_completed__now) def test_create_background_status__invalid_last_completed__events_queue(self): save_mock_bgjob(JournalCSVBackgroundTask.__action__, queue_id=constants.BGJOB_QUEUE_ID_EVENTS, status=constants.BGJOB_STATUS_COMPLETE, ) status_dict = background_task_status.create_background_status() assert not is_stable(status_dict['status']) > self.assert_unstable_dict(status_dict['queues'].get('events', {})) test_background_task_status.py:116: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ val = {'err_msgs': [], 'errors': {'application_autochecks': {'err_msgs': [], 'in_monitoring_period': 0, 'status': 'stable', ...table'}, 'ingest_articles': {'err_msgs': [], 'last_run': None, 'last_run_status': None, 'status': 'stable'}, ...}, ...} @staticmethod def assert_unstable_dict(val): > assert not is_stable(val.get('status')) E AssertionError: assert not True E + where True = is_stable('stable') E + where 'stable' = <built-in method get of dict object at 0x165c907c0>('status') E + where <built-in method get of dict object at 0x165c907c0> = {'err_msgs': [], 'errors': {'application_autochecks': {'err_msgs': [], 'in_monitoring_period': 0, 'status': 'stable', 'total': 0}, 'article_bulk_create': {'err_msgs': [], 'in_monitoring_period': 0, 'status': 'stable', 'total': 0}, 'article_bulk_delete': {'err_msgs': [], 'in_monitoring_period': 0, 'status': 'stable', 'total': 0}, 'ingest_articles': {'err_msgs': [], 'in_monitoring_period': 0, 'status': 'stable', 'total': 0}, ...}, 'last_completed_job': '2024-11-04T04:46:47Z', 'last_run_successful': {'application_autochecks': {'err_msgs': [], 'last_run': None, 'last_run_status': None, 'status': 'stable'}, 'article_bulk_create': {'err_msgs': [], 'last_run': None, 'last_run_status': None, 'status': 'stable'}, 'article_bulk_delete': {'err_msgs': [], 'last_run': None, 'last_run_status': None, 'status': 'stable'}, 'ingest_articles': {'err_msgs': [], 'last_run': None, 'last_run_status': None, 'status': 'stable'}, ...}, ...}.get test_background_task_status.py:104: AssertionError

doajtest/unit/test_background_task_status.py:232 (TestBackgroundTaskStatus.test_create_background_status__queued_valid_oldest) self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__queued_valid_oldest> @apply_test_case_config(bg_monitor_queued_config__a) def test_create_background_status__queued_valid_oldest(self): save_mock_bgjob(JournalCSVBackgroundTask.__action__, status=constants.BGJOB_STATUS_QUEUED, ) status_dict = background_task_status.create_background_status() print(json.dumps(status_dict, indent=4)) journal_csv_dict = status_dict['queues']['scheduled_short']['queued'].get(JournalCSVBackgroundTask.__action__, {}) > assert is_stable(status_dict['status']) E AssertionError: assert False E + where False = is_stable('unstable') test_background_task_status.py:243: AssertionError

doajtest/unit/test_background_task_status.py:205 (TestBackgroundTaskStatus.test_create_background_status__queued_valid_total) self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__queued_valid_total> @apply_test_case_config(bg_monitor_queued_config__zero_total) def test_create_background_status__queued_valid_total(self): save_mock_bgjob(JournalCSVBackgroundTask.__action__, status=constants.BGJOB_STATUS_COMPLETE, ) status_dict = background_task_status.create_background_status() journal_csv_dict = status_dict['queues']['scheduled_short']['queued'].get(JournalCSVBackgroundTask.__action__, {}) > assert is_stable(status_dict['status']) E AssertionError: assert False E + where False = is_stable('unstable') test_background_task_status.py:215: AssertionError

doajtest/unit/test_background_task_status.py:130 (TestBackgroundTaskStatus.test_create_background_status__valid_last_completed) self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__valid_last_completed> @apply_test_case_config(bg_monitor_last_completed__a) def test_create_background_status__valid_last_completed(self): save_mock_bgjob(JournalCSVBackgroundTask.__action__, queue_id=constants.BGJOB_QUEUE_ID_EVENTS, status=constants.BGJOB_STATUS_COMPLETE, ) save_mock_bgjob(AnonExportBackgroundTask.__action__, queue_id=constants.BGJOB_QUEUE_ID_SCHEDULED_LONG, status=constants.BGJOB_STATUS_COMPLETE, ) status_dict = background_task_status.create_background_status() > assert is_stable(status_dict['status']) E AssertionError: assert False E + where False = is_stable('unstable') test_background_task_status.py:142: AssertionError

doajtest/unit/test_background_task_status.py:145 (TestBackgroundTaskStatus.test_create_background_status__valid_last_completed__no_record) self = <doajtest.unit.test_background_task_status.TestBackgroundTaskStatus testMethod=test_create_background_status__valid_last_completed__no_record> @apply_test_case_config(bg_monitor_last_completed__now) def test_create_background_status__valid_last_completed__no_record(self): status_dict = background_task_status.create_background_status() > assert is_stable(status_dict['status']) E AssertionError: assert False E + where False = is_stable('unstable') test_background_task_status.py:149: AssertionError

@RK206 The testing was a bit confused now there is an additional monitoring stream, so I have replaced it with a parameterised test that gets all the variations and that appears to be working fine.

RK206 · 2024-11-04T05:08:32Z

portality/bll/services/background_task_status.py

@@ -4,7 +4,9 @@
 import itertools
 from typing import Iterable

-from portality.constants import BGJOB_QUEUE_ID_LONG, BGJOB_QUEUE_ID_MAIN, BGJOB_STATUS_ERROR, BGJOB_STATUS_QUEUED, \
+import constants


should it import from portality?

RK206 · 2024-11-04T05:09:05Z

portality/bll/services/background_task_status.py

@@ -4,7 +4,9 @@
 import itertools
 from typing import Iterable

-from portality.constants import BGJOB_QUEUE_ID_LONG, BGJOB_QUEUE_ID_MAIN, BGJOB_STATUS_ERROR, BGJOB_STATUS_QUEUED, \
+import constants
+from constants import BGJOB_STATUS_COMPLETE


same as above. import from portality?

Ah, that's a pycharm thing, it sometimes does the imports for you, and it does them relative, which seems to work locally but not anywhere else. Very annoying!

…itoring

…he background status variations

Steven-Eardley

A few changes for @RK206 due to recent deployment changes.

Note the queues have been added in addition to the old ones (main and long_running) so we don't need to touch the old tasks, and we only need to add the symlinks for deployment for the new consumers. The release will therefore be fairly straightforward.

Steven-Eardley · 2024-11-28T08:32:13Z

deploy/supervisor/production-background/huey-events.conf

@@ -0,0 +1,9 @@
+[program:huey-events]
+command=/home/cloo/doaj/bin/python /home/cloo/doaj/bin/huey_consumer.py -v portality.tasks.consumer_events_queue.events_queue


updated deployment - changed binary paths /home/cloo/doaj/**venv/**bin/python and /home/cloo/doaj/**venv/**bin/huey_consumer.py

Steven-Eardley · 2024-11-28T08:32:36Z

deploy/supervisor/production-background/huey-scheduled-long.conf

@@ -0,0 +1,9 @@
+[program:huey-scheduled-long]
+command=/home/cloo/doaj/bin/python /home/cloo/doaj/bin/huey_consumer.py -v portality.tasks.consumer_scheduled_long_queue.scheduled_long_queue


venv path again

Steven-Eardley · 2024-11-28T08:32:47Z

deploy/supervisor/production-background/huey-scheduled-short.conf

@@ -0,0 +1,9 @@
+[program:huey-scheduled-short]
+command=/home/cloo/doaj/bin/python /home/cloo/doaj/bin/huey_consumer.py -v portality.tasks.consumer_scheduled_short_queue.scheduled_short_queue


Steven-Eardley · 2024-11-28T08:34:30Z

doajtest/helpers.py

-        long_running.always_eager = True
-        main_queue.immediate = True
-        long_running.immediate = True
+        events_queue.always_eager = True


syntax change for new huey - always_eager is now immediate

Steven-Eardley · 2024-11-28T09:01:14Z

I retract my suggestion for @RK206 to make the above changes for new deployment - I'll take care of that and fix the conflicts

richard-jones added 4 commits October 10, 2024 10:45

update bg monitoring settings for mission critical monitoring changes

2d2a500

introduce new queues and assign the jobs to the appropriate ones, and…

2584b0c

… also resolve merge conflict

implement last_successful_run code and configuration for status

85ebff1

implement last run successful tests and related fixes

c6f7a11

richard-jones requested review from RK206 and Steven-Eardley October 11, 2024 10:19

richard-jones mentioned this pull request Oct 11, 2024

update bg monitoring settings for mission critical monitoring changes #2419

Closed

22 tasks

RK206 requested changes Nov 4, 2024

View reviewed changes

richard-jones added 2 commits November 12, 2024 10:55

start of transition to parameterised testing for background tasks mon…

46a19b4

…itoring

update testing to use parameterised tests and exhaustively test all t…

5cf7bf0

…he background status variations

richard-jones requested a review from RK206 November 14, 2024 13:13

RK206 approved these changes Nov 15, 2024

View reviewed changes

Steven-Eardley approved these changes Nov 28, 2024

View reviewed changes

Steven-Eardley merged commit 9fd67b1 into develop Nov 28, 2024
1 of 2 checks passed

Steven-Eardley deleted the feature/3959_bg_monitoring_fixed branch November 28, 2024 16:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/3959 bg monitoring fixed #2423

Feature/3959 bg monitoring fixed #2423

richard-jones commented Oct 11, 2024

RK206 left a comment

RK206 Nov 4, 2024

richard-jones Nov 14, 2024

RK206 Nov 4, 2024

RK206 Nov 4, 2024

richard-jones Nov 11, 2024

Steven-Eardley left a comment

Steven-Eardley Nov 28, 2024

Steven-Eardley Nov 28, 2024

Steven-Eardley Nov 28, 2024

Steven-Eardley Nov 28, 2024

Steven-Eardley commented Nov 28, 2024

		@@ -0,0 +1,9 @@
		[program:huey-events]
		command=/home/cloo/doaj/bin/python /home/cloo/doaj/bin/huey_consumer.py -v portality.tasks.consumer_events_queue.events_queue

		@@ -0,0 +1,9 @@
		[program:huey-scheduled-long]
		command=/home/cloo/doaj/bin/python /home/cloo/doaj/bin/huey_consumer.py -v portality.tasks.consumer_scheduled_long_queue.scheduled_long_queue

		@@ -0,0 +1,9 @@
		[program:huey-scheduled-short]
		command=/home/cloo/doaj/bin/python /home/cloo/doaj/bin/huey_consumer.py -v portality.tasks.consumer_scheduled_short_queue.scheduled_short_queue

Feature/3959 bg monitoring fixed #2423

Feature/3959 bg monitoring fixed #2423

Conversation

richard-jones commented Oct 11, 2024

Replaces PR: #2419

update bg monitoring settings for mission critical monitoring changes

Developer Checklist

Reviewer Checklist

Testing

Deployment

Configuration changes

Monitoring

RK206 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Steven-Eardley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Steven-Eardley commented Nov 28, 2024