Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabling debug logs for rename perf test #2509

Draft
wants to merge 16 commits into
base: master
Choose a base branch
from

Conversation

anushka567
Copy link
Member

@anushka567 anushka567 commented Sep 18, 2024

Description

Enabled debug logs for rename perf test

Link to the issue in case of a bug fix.

NA

Testing details

  1. Manual - Ran on actual test data. Working as intended.
  2. Unit tests - NA
  3. Integration tests - NA

@anushka567 anushka567 requested review from Tulsishah and a team as code owners September 18, 2024 09:32
@kislaykishore kislaykishore requested review from a team, BrennaEpp, kislaykishore and gargnitingoogle and removed request for a team, BrennaEpp and kislaykishore September 18, 2024 09:33
@anushka567 anushka567 force-pushed the hns-rename-benchmark-upgrade-gcloud branch from 3b8989b to a3c5ced Compare September 18, 2024 09:43
@kislaykishore kislaykishore requested a review from a team September 18, 2024 10:04
@anushka567 anushka567 marked this pull request as draft September 18, 2024 12:56
@anushka567 anushka567 changed the title Upgrading gcloud version for rename perf tests Enabling debug logs for rename perf test Sep 18, 2024
perfmetrics/scripts/continuous_test/gcp_ubuntu/build.sh Outdated Show resolved Hide resolved
else :
# Creating config file for mounting with hns disabled.
with open("/tmp/config.yml",'w') as mount_config:
mount_config.write("enable-hns: false")
mount_flags = "--config-file=/tmp/config.yml --implicit-dirs --rename-dir-limit=1000000 --stackdriver-export-interval=30s"
mount_flags = "--config-file=/tmp/config.yml --log-severity=trace --log-format \"text\" --log-file {} --implicit-dirs --rename-dir-limit=1000000 --stackdriver-export-interval=30s".format(LOG_FILE)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding trace-logging affect the performance of the test run. Is that accounted for in thresholds ?

@@ -133,7 +133,7 @@ def test_perform_testing_flat(self, mock_log, mock_record_time_of_operation,
test_type = "flat"
num_samples = 4
results = {}
mount_flags = "--config-file=/tmp/config.yml --implicit-dirs --rename-dir-limit=1000000 --stackdriver-export-interval=30s"
mount_flags = "--config-file=/tmp/config.yml --log-severity=trace --log-format \"text\" --log-file /tmp/gcsfuse-logs-rename.txt --implicit-dirs --rename-dir-limit=1000000 --stackdriver-export-interval=30s"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question as above about the performance issue with log-severity trace.

perfmetrics/scripts/upgrade_gcloud.sh Show resolved Hide resolved
export PATH=/usr/local/google-cloud-sdk/bin:$PATH
echo 'export PATH=/usr/local/google-cloud-sdk/bin:$PATH' >> ~/.bashrc
gcloud version && rm gcloud.tar.gz
sudo /usr/local/google-cloud-sdk/bin/gcloud components update
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need the full path to gcloud here as we have already update the PATH variable locally.

Suggested change
sudo /usr/local/google-cloud-sdk/bin/gcloud components update
sudo gcloud components update

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While trying to do this, I was facing this issue

ERROR: (gcloud.components.update) You cannot perform this action because this Google Cloud CLI installation is managed by an external package manager.
Please consider using a separate installation of the Google Cloud CLI created through the default mechanism described at: https://cloud.google.com/sdk/

but this did not come up when using sudo /usr/local/google-cloud-sdk/bin/gcloud components update

@anushka567 anushka567 force-pushed the hns-rename-benchmark-upgrade-gcloud branch from a3c5ced to b79a9c9 Compare September 19, 2024 05:17
Copy link

codecov bot commented Sep 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.90%. Comparing base (b93c72b) to head (117db8d).
Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2509      +/-   ##
==========================================
+ Coverage   78.85%   78.90%   +0.05%     
==========================================
  Files         106      106              
  Lines       11686    11686              
==========================================
+ Hits         9215     9221       +6     
+ Misses       1992     1987       -5     
+ Partials      479      478       -1     
Flag Coverage Δ
unittests 78.90% <ø> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@anushka567 anushka567 force-pushed the hns-rename-benchmark-upgrade-gcloud branch from ddf6c78 to 117db8d Compare September 19, 2024 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants