Multiple collectors #204
Open
Multiple collectors #204
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ne time. Most unix systems limit the number of file handles any one process can have open at a time to 1024. When the collector gets behind, it may try to open too many files for completed jobs at once. Limit this with a new collector setting which defaults to 300. They'll be picked up once the ones polled are processed. This commit adds App::Yath::Options::Collector
* Added another collector option * Renamed collector option * Put option in test and start commands * Make warning when job limit is hit a proper event * Only issue the warning once per test run * Use the correct settings file in collector
Primarily a typo, but also a race condition that could have collector exit with jobs still waiting to send events.
I am too tired right now after finally getting this working. It needs review, and I will probably want to clean it up myself after some sleep.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I need to clean this up a bit, but would love early review.
@toddr and @atoomic This is based off the updated PR for too many open handles. I was hoping you could pull this down and give it a try, maybe set
--max-open-jobsto a lowish number like 100, and then also add--spawn-worker-at-maxto see if that solves your problem without the collector(s) getting horribly backed-up. If you are in a situation where the collector is THAT backed up then your tests may finish well before yath does which would cost you extra time, assuming your max open handles limit is per-process and not per-user extra collectors can help work though the backlog.This method does not limit the number of collectors, rather whenever a collector hits the max open files it forks, the fork finishes up all the jobs already started leaving the main collector to grab new jobs, once all the jobs of the forked collector finishes it exits, if another backup happens a new one forks. If you have multiple backups you get multiple collectors. So for example setting -j100, and setting --max-open-jobs to 1 would mean 100 collectors get spawned (yikes! do not do that!) it could actually climb higher than 100 if the collectors take too long even on a single job. So you really do want a decently high --max-open-jobs.
Like all things, you need to balance the 3 collector settings and tune them to something that works right for your setup, which is why I wanted all these to be configurable.