Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FDRS: Fetch only the latest year related data #2280

Merged
merged 3 commits into from
Nov 5, 2024

Conversation

susilnem
Copy link
Member

@susilnem susilnem commented Oct 3, 2024

Addresses

Changes

  • KPI_noLocalUnits changed to KPI_noBranches
  • Updated the cron job databank -> FDRS
    This pull request's related to databank/management/commands/sources/FDRS.py.
  • Change the field name in fdrs and worldbank related fields.
  • Add new migrations file.

Need to run these commands.

python manage.py ingest_databank and python manage.py ingest_worldbank

Improvements to data fetching and processing:

  • Changed the key from KPI_noLocalUnits to KPI_noBranches.

  • Updated the prefetch function to fetch the latest data for each indicator by using the max function to get the entry with the highest year. (databank/management/commands/sources/FDRS.py)

  • Modified the load function to ensure only the latest year's available data is used by checking the year before setting the value. (databank/management/commands/sources/FDRS.py)

Checklist

Things that should succeed before merging.

  • Updated/ran unit tests
  • Updated CHANGELOG.md

Release

If there is a version update, make sure to tag the repository with the latest version.

databank/management/commands/sources/FDRS.py Outdated Show resolved Hide resolved
databank/management/commands/sources/FDRS.py Outdated Show resolved Hide resolved
@susilnem susilnem force-pushed the fix/fdrs-latest-year-data branch 2 times, most recently from dda3075 to 45a6f6a Compare October 7, 2024 14:39
Copy link
Member

@thenav56 thenav56 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's wait until approval from QA

@susilnem susilnem marked this pull request as ready for review October 24, 2024 10:12
@szabozoltan69
Copy link
Contributor

Should we merge this?

@szabozoltan69 szabozoltan69 merged commit 8117b57 into develop Nov 5, 2024
2 checks passed
@szabozoltan69 szabozoltan69 deleted the fix/fdrs-latest-year-data branch November 5, 2024 12:12
@susilnem
Copy link
Member Author

susilnem commented Nov 5, 2024

@szabozoltan69
we also need to run these commands:
python manage.py ingest_databank and python manage.py ingest_worldbank

@szabozoltan69
Copy link
Contributor

Both were run fine (ingest_databank had a slight issue).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants