Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

As a developer, I want to reduce code issues revealed through static analysis #276

Open
epag opened this issue Aug 21, 2024 · 35 comments
Open

Comments

@epag
Copy link
Collaborator

epag commented Aug 21, 2024


Author Name: James (James)
Original Redmine Issue: 96440, https://vlab.noaa.gov/redmine/issues/96440
Original Date: 2021-09-20
Original Assignee: Evan


Given a static analysis of the codebase performed with SonarQube or similar
When I see a lot of code smells, bugs, vulnerabilities etc.
Then I want to review and eliminate some of 'em


Redmine related issue(s): 96459, 112924


@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T10:47:45Z


Here's the mile-high view from SQ as of commit:wres|e955b3533d16b2f8f68ee392d0b117f05541bdbd:

!sq_top.png!

Not sure what is wrong with the code coverage metric, perhaps sq and other tooling not communicating, but that is obviously wrong.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T10:49:23Z


!sq_vulns.png!

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T10:50:40Z


!bugs.png!

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T10:51:45Z


!reliability.png!

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T10:55:04Z


!security.png!

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T10:55:49Z


Reminded again that sq is really a neat tool.

Obviously, some of these will be debatable or false positives, but a lot won't be.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T10:56:06Z


Some of them are probably urgent.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T10:59:17Z


Approaching 100k sloc too, sheesh.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T12:27:39Z


As an aside, this is probably something we should add to our jenkins build.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T12:47:59Z


James wrote:

Not sure what is wrong with the code coverage metric, perhaps sq and other tooling not communicating, but that is obviously wrong.

https://community.sonarsource.com/t/sonarqube-community-version-8-3-is-showing-0-code-coverage-however-with-the-same-configurations-i-am-able-to-get-coverage-in-community-version-8-1/28007

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: Jesse (Jesse)
Original Date: 2021-09-20T14:13:27Z


The ticket for the capability on the VLab side: #66414

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T14:24:15Z


Jesse wrote:

The ticket for the capability on the VLab side: #66414

Thanks. Doesn't hurt to link, I suppose, so I did.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T16:54:27Z


Hadn't intended to start addressing these, but I guess I did. Sunk a couple of hours in. Good task between tasks.

!sq_top2.png!

A lot of the security suggestions are false positives.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2021-09-20T17:18:48Z


Progress in commit:wres|d93a710e22c7d5acfb335b4a91c49c4fe0a59c1c.

Again, the sort of ticket that anyone can increment between tasks.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-02-24T10:23:53Z


Pushing a few more against this one.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-02-24T10:49:03Z


commit:wres|054bc754d66a2799049aaca16ff76d877fed1944

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-02-28T10:42:14Z


In preparing for some work on #59791, first fixing a bajillion code smells in @wres-io@ that have been hanging around forever. The lack of attention to detail in parts of that code base is frustrating, to say the least.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-02-28T11:20:16Z


I see a different spelling of "bankful" (bankfull) in different WRDS APIs. I wonder if this is a typo on our end (and, if so, a bug, since Jackson maps on names) or if the APIs do vary...

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-02-28T14:24:15Z


About to push a lot of small changes. Some of these involve changes to Jackson annotations to deserialize json or waterml into POJOs, so we should also do some testing of all WRDS services, including thresholds, before deploying as I'm not sure how good the unit test coverage is for some of these classes.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-02-28T14:27:06Z


Painful number of fixes here.

commit:wres|522ad907ec14da7e17e5995d5366818f610e5774

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-02-28T14:28:06Z


You'll probably want to pull at your earliest convenience.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-02-28T14:43:42Z


There are still quite a few code smells throughout the codebase, but stopping there for now. This is one that we'll keep returning to.

Good practice is to avoid these smells by installing static analysis tools to identify them as you develop, but I appreciate that there are some restrictions on OWP developers that make their lives harder, so this may not always be easy/possible.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2023-02-28T14:57:44Z


James,

I'll try to deploy to staging later today so that I can do some early testing of WRDS services. Thanks for the heads up,

Hank

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2023-03-06T18:50:21Z


James wrote:

About to push a lot of small changes. Some of these involve changes to Jackson annotations to deserialize json or waterml into POJOs, so we should also do some testing of all WRDS services, including thresholds, before deploying as I'm not sure how good the unit test coverage is for some of these classes.

Doing some initial testing using 20230306-ff9d924. The following tests all used an SA executed on nwcal-wres-ti01 (Docker wrapped JDK image) and forecast period 3/1/2023 - 3/5/2023:

  • AHPS forecasts from WRDS against WRDS observations for two features: succeeds with reasonable results.
  • NWM forecasts from WRDS against USGS observations with WRDS supplied NWS thresholds for two features: succeeds with reasonable results.
  • NWM forecasets from D-Store against USGS observations: succeeds with reasonable results.
  • AHPS forecasts from WRDS against WRDS observations for an RFC identified via WRDS location service: This evaluation is on-going, since it involves all of MARFC.

So far so good,

Hank

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2023-03-06T20:06:41Z


The last evaluation using using the connection failed due to a disconnection. It was probably just a network hiccup.

It certainly appears that the WRES successfully communicated with WRDS, given it reached the point of data acquisition, but I would like to see the evaluation complete. I'll login and try again, this time running the evaluation in a @nohup@.

Hank

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: Hank (Hank)
Original Date: 2023-03-06T21:10:42Z


The evaluation succeeded in 25m 37s. Good.

For now, my testing is done. I'll leave the ticket on-hold as a reminder to repeat these tests when we deploy.

Hank

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-04-03T10:20:52Z


Spotted a bunch of additional issues related to declaration helpers, which I'm going to resolve alongside #113677.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-04-03T11:20:02Z


commit:wres|f03f8bd65cf2e86148374ea59af3d918f16678c0

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-04-03T12:25:10Z


As noted in the @build.gradle@:

    // The liquibase changesets are in the root project(!?), give wres-io access
    // to those changesets on the test time classpath.

Going to try and fix this because it results in an annoying duplication warning at build time in the IDE from these files being duplicated on the cp.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-04-03T14:12:29Z


James wrote:

As noted in the @build.gradle@:

[...]

Going to try and fix this because it results in an annoying duplication warning at build time in the IDE from these files being duplicated on the cp.

Substantially fixed in commit:wres|12488e4e3b7719c9793762f8e4472ce7bfcd9f1a, although that contained a minor platform-dependent path error that was fixed in commit:wres|b13a758a1fdb7c9afae8d7f021f41578d952288a.

To the backlog again.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-04-24T10:20:57Z


Before starting some major refactoring for #113677, taking an opportunity to rename/relocate some things that are potentially confusing to new developers. Not identified by static analysis, as such, but this ticket is close enough.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-04-24T10:29:30Z


James wrote:

Before starting some major refactoring for #113677, taking an opportunity to rename/relocate some things that are potentially confusing to new developers. Not identified by static analysis, as such, but this ticket is close enough.

Some of the things I'm looking at are also connected to #59791 - will tag that ticket in the relevant commits too.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-04-24T10:40:43Z


First thing I want to do is to place everything related to reading from WRDS in one place. We have (WRDS) readers for features, thresholds and time-series data and they are variously distributed throughout @wres-io@. Since we have a @wres.io.reading.wrds@, it makes sense to place everything related to reading from WRDS in that package.

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-04-24T11:54:21Z


commit:wres|e3db852fe7f3f7811902504152b1c86a222f41a3
commit:wres|445b7c159277396f9a641d367156839818099172

@epag
Copy link
Collaborator Author

epag commented Aug 21, 2024


Original Redmine Comment
Author Name: James (James)
Original Date: 2023-04-24T15:06:35Z


Done with this (for now) as of commit:wres|76c53c091e08eaa8fef480ea36e0eaa58a0a2141.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant