-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LIU-383: Initial work towards linter workflows #266
Changes from all commits
348cacf
b4340ab
4eec538
ecbf3aa
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,3 +1,4 @@ | ||
[Main] | ||
disable=all | ||
disable=C, R, W, no-name-in-module, c-extension-no-member, no-member, import-error, unsupported-membership-test | ||
enable=logging-not-lazy,logging-format-interpolation | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -82,7 +82,7 @@ def __init__(self, init_dict=None): | |
self.update(init_dict) | ||
if "oid" not in self: | ||
self.update({"oid": None}) | ||
return super().__init_subclass__() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. question: Removed return statement The return statement has been removed from the init_subclass method. Ensure that this change does not affect the behavior of the method. |
||
super().__init_subclass__() | ||
|
||
def _addSomething(self, other, key, name=None): | ||
if key not in self: | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -125,7 +125,7 @@ def accumulate_pgt_unroll_drop_data(drop: dict): | |
} | ||
if drop["reprodata"].get("rmode") is None: | ||
level = REPRO_DEFAULT | ||
drop["reprodata"]["rmode"] = str(level.level) | ||
drop["reprodata"]["rmode"] = str(level) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. question: Changed level to string The level is now being converted to a string directly. Ensure that this change is intentional and that level is always convertible to a string. |
||
drop["reprodata"][level.name] = {} | ||
else: | ||
level = rflag_caster(drop["reprodata"]["rmode"]) | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -139,7 +139,7 @@ def __exit__(self, typ, value, traceback): | |
|
||
def _get_json(self, url): | ||
ret = self._GET(url) | ||
return json.load(ret) if ret else None | ||
return json.load(ret) if ret else {} | ||
myxie marked this conversation as resolved.
Show resolved
Hide resolved
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I have updated this to return a dictionary as we were not checking for |
||
|
||
def _post_form(self, url, content=None): | ||
if content is not None: | ||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -134,7 +134,7 @@ def setCompleted(self): | |||||
|
||||||
@property | ||||||
def dataURL(self) -> str: | ||||||
return "ngas://%s:%d/%s" % (self.ngasSrv, self.ngasPort, self.fileId) | ||||||
return "ngas://%s:%s/%s" % (self.ngasSrv, self.ngasPort, self.fileId) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. suggestion (code-quality): Replace interpolated string formatting with f-string (
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This can be taken care of another day when addressing Pylint's warnings . |
||||||
|
||||||
# Override | ||||||
def generate_reproduce_data(self): | ||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -133,7 +133,7 @@ def getIO(self) -> DataIO: | |
# self.mapped_inputs = identify_named_ports( | ||
# self._producers, {}, self.keyargs, mode="inputs" | ||
# ) | ||
logger.debug("Parameters found: {}", self.parameters) | ||
logger.debug("Parameters found: %s", self.parameters) | ||
myxie marked this conversation as resolved.
Show resolved
Hide resolved
|
||
return S3IO( | ||
self.aws_access_key_id, | ||
self.aws_secret_access_key, | ||
|
@@ -370,7 +370,7 @@ def _exists(self) -> Tuple[bool, bool]: | |
logger.info("Object: %s does not exist", self._key) | ||
return True, False | ||
else: | ||
raise ErrorIO() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. question: Changed exception type The exception type has been changed from ErrorIO to RuntimeError with a message. Ensure that this change is intentional and that RuntimeError is the appropriate exception type for this scenario. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
raise RuntimeError("Error occured in Client: %s", e.response) | ||
|
||
@overrides | ||
def exists(self) -> bool: | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -291,6 +291,7 @@ def buffer(self) -> memoryview: | |
return self._buf.getbuffer() | ||
|
||
|
||
# pylint: disable=possibly-used-before-assignment | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. question: Pylint directive added The pylint directive to disable possibly-used-before-assignment has been added. Ensure that this is necessary and does not hide potential issues. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is necessary due to the Python >= 3.8 restriction on shared memory functionality, which means we end up with a |
||
class SharedMemoryIO(DataIO): | ||
""" | ||
A DataIO class that writes to a shared memory buffer | ||
|
@@ -317,10 +318,10 @@ def _write(self, data, **kwargs) -> int: | |
total_size = len(data) + self._written | ||
if total_size > self._buf.size: | ||
self._buf.resize(total_size) | ||
self._buf.buf[self._written : total_size] = data | ||
self._buf.buf[self._written: total_size] = data | ||
self._written = total_size | ||
else: | ||
self._buf.buf[self._written : total_size] = data | ||
self._buf.buf[self._written: total_size] = data | ||
self._written = total_size | ||
self._buf.resize(total_size) | ||
# It may be inefficient to resize many times, but assuming data is written 'once' this is | ||
|
@@ -357,6 +358,7 @@ def exists(self) -> bool: | |
def delete(self): | ||
self._close() | ||
|
||
# pylint: enable=possibly-used-before-assignment | ||
myxie marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
class FileIO(DataIO): | ||
""" | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -383,8 +383,8 @@ def submit_and_monitor_pgt(self): | |
""" | ||
Combines submission and monitoring steps of a pgt. | ||
""" | ||
session_id = self.submit_pgt() | ||
monitoring_thread = self._monitor(session_id) | ||
self.submit_pgt() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
monitoring_thread = self._monitor() | ||
myxie marked this conversation as resolved.
Show resolved
Hide resolved
|
||
monitoring_thread.join() | ||
|
||
def submit_pg(self): | ||
|
@@ -408,6 +408,6 @@ def submit_and_monitor_pg(self): | |
""" | ||
Combines submission and monitoring steps of a pg. | ||
""" | ||
session_id = self.submit_pg() | ||
monitoring_thread = self._monitor(session_id) | ||
self.submit_pg() | ||
monitoring_thread = self._monitor() | ||
monitoring_thread.join() |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -408,7 +408,7 @@ def get_param_value(attr_name, default_value): | |
has_app_param = ( | ||
"applicationArgs" in kwargs and attr_name in kwargs["applicationArgs"] | ||
) | ||
|
||
param = default_value | ||
myxie marked this conversation as resolved.
Show resolved
Hide resolved
|
||
if has_component_param and has_app_param: | ||
logger.warning( | ||
f"Drop has both component and app param {attr_name}. Using component param." | ||
|
@@ -428,8 +428,6 @@ def get_param_value(attr_name, default_value): | |
pass | ||
else: | ||
param = kwargs["applicationArgs"].get(attr_name).value | ||
else: | ||
param = default_value | ||
Comment on lines
-431
to
-432
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. question: Removed else block The else block that assigns default_value to param has been removed. Ensure that this change does not affect the logic where param should be assigned default_value if no other conditions are met. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We don't need this any more now that we are initialising the |
||
return param | ||
|
||
# Take a class dlg defined parameter class attribute and create an instanced attribute on object | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,6 +5,8 @@ | |
To run it standalone, change the directories, which are now hardcoded | ||
""" | ||
|
||
# pylint: skip-file | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am skipping this file completely because it is using |
||
import datetime | ||
import os | ||
import time | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These settings mean Pylint will return an error code of 0 unless we are either under 9 or fail on Errors.
We should be at a 10.0 currently, and should have no Errors after this PR. What these settings also allow is for us to enable Warnings (thus seeing them introduced in the code), without us Failing any tests. This means we can, if we want to, enable Warnings at some point whilst we address them incrementally, without it affecting the CI. Something to think about moving forward I expect.
We score a 9.47(?) with the Warnings enabled, so it's unlikely we'd ever go below 9. It is necessary to get the
--fail-on=E
to work properly, however.