Recycle actor heap chunks after GC instead of returning to pool #4531
Conversation
Before this commit, any unused chunks after actor heap garbage collection would be destroyed and returned to the memory pool immediately for reuse by the runtime or any actor. This commit changes things so that instead of destroying and returning the chunks immediatelly, we assume the actor will likely need more memory as it runs more behaviors and keep the recently unused chunks around in case that happens. This is generally more efficient than destroying a chunk and getting a new one from the memory pool because both destorying a chunk and allocating a new one involve updating the pagemap for the chunk to indicate which actor owns the chunk. Updating the pagemap is an expensive operation which we can avoid if we recycle the chunks instead. The main drawback is that since actors will no longer return chunks to the memory pool immediately after a GC, the overall system might end up using more memory as any freed chunks can only be reused by the actor that owns them and the runtime and other actors can no longer reuse that memory as they previously might have been able to.
ponylang-main
added
the
discuss during sync
| small_chunk_t* n = NULL; | ||
|
|
||
| // recycle a small chunk if available because it avoids setting the pagemap | ||
| if (NULL != heap->small_recyclable) |
There was a problem hiding this comment.
our normal pattern in the codebase is to compare to NULL as the second item. Is there a reason for the variance in this patch?
There was a problem hiding this comment.
no reason except that i accidentally forgot an = and didn't notice and that's much easier to track down with the condition written this way..
|
I see this will free any recycled chunks that aren't reused after 1 gc pass. Can you help me work through how this will work in practice. If an actor never gets gc'd, this won't have any impact. If an actor only got gc'd once, then some unknown amount of memory would not be freed, and if the actor is gc'd more than once, the same would still apply, some amount of memory would continue to be held for recycling. Yes? |
correct..
that last point is where there's possibility of tweaking things by having the actor |
SeanTAllen
added
the
changelog - changed
|
Hi @dipinhora, The changelog - changed label was added to this pull request; all PRs with a changelog label need to have release notes included as part of the PR. If you haven't added release notes already, please do. Release notes are added by creating a uniquely named file in the The basic format of the release notes (using markdown) should be: Thanks. |
|
release notes added |
ponylang-main
removed
the
discuss during sync
|
@dipinhora first night after this was merged, all the stress tests failed. I'm waiting to see what happens with tonight's. |
ponylang-main
added
the
discuss during sync
@SeanTAllen i looked through all the logs and they all end with: it doesn't seem like anything actionable (i.e. no crashes). |
|
scratch that... looking at older stress test runs they all finished in under 30 mins - 1 hour and the new runs seem to time out after 6 hours.. something too look into.. |
The implementation of actor heap large chunk recycling from ponylang#4531 too naive and results in actors wasting huge amounts of time related to large chunk recycling. This commit effectively disables the large chunk recycling but doesn't undo the code changes at the moment because the expectation is that large chunk recycling will be re-enabled in the near future with an improved implementation.
|
@SeanTAllen #4534 has been opened to resolve the stress test issue. |
The implementation of actor heap large chunk recycling from #4531 too naive and results in actors wasting huge amounts of time related to large chunk recycling. This commit effectively disables the large chunk recycling but doesn't undo the code changes at the moment because the expectation is that large chunk recycling will be re-enabled in the near future with an improved implementation.
The implementation of actor heap chunk recycling from ponylang#4531 had two bugs. First, the large heap re-use logic (which was temporarily disabled in ponylang#4534) had a bug related to how it updated the large chunk recyclable list pointer in the heap. Second, the memory clearing logic in the `ponyint_heap_endgc` function was clearing more of the heap than it should have been resulting in a memory leak for both small and large chunk recyclable chunks. This commit re-enabled large chunk recycling (undoing ponylang#4534) and fixes both bugs so that both large chunk and small chunk recycling work as expected without memory leaks.
The implementation of actor heap chunk recycling from #4531 had two bugs. First, the large heap re-use logic (which was temporarily disabled in #4534) had a bug related to how it updated the large chunk recyclable list pointer in the heap. Second, the memory clearing logic in the `ponyint_heap_endgc` function was clearing more of the heap than it should have been resulting in a memory leak for both small and large chunk recyclable chunks. This commit re-enabled large chunk recycling (undoing #4534) and fixes both bugs so that both large chunk and small chunk recycling work as expected without memory leaks.
Before this commit, any unused chunks after actor heap garbage collection would be destroyed and returned to the memory pool immediately for reuse by the runtime or any actor.
This commit changes things so that instead of destroying and returning the chunks immediatelly, we assume the actor will likely need more memory as it runs more behaviors and keep the recently unused chunks around in case that happens. This is generally more efficient than destroying a chunk and getting a new one from the memory pool because both destorying a chunk and allocating a new one involve updating the pagemap for the chunk to indicate which actor owns the chunk. Updating the pagemap is an expensive operation which we can avoid if we recycle the chunks instead. The main drawback is that since actors will no longer return chunks to the memory pool immediately after a GC, the overall system might end up using more memory as any freed chunks can only be reused by the actor that owns them and the runtime and other actors can no longer reuse that memory as they previously might have been able to.