Improve logic for restarting kernels with new ports during initial startup

[skukhtichev]

The proposal defines the logic in restarter.py for setting _initial_startup value to False only when the kernel sends kernel_info reply. Introduced changes solve issues described in #347. The introduced changes allow to identify issues with shell channel (e.g. due to the ZMQError: Address already in use issue)
The proposal also defines logic for notifying client (Notebook Server, Kernel gateway, etc.) that the kernel was restarted with new ports.
The Proposal includes the following changes:

• Add the logic for monitoring shell channel:
• Configurable startup_time parameter is introduced. This parameter defines the timeout for kernel_reply response. Default value is set to 0 (kernel_info reply monitoring disables). It is disabled by default, because different kernels (e.g. kernels with spark context) require different time before the kernel will be ready.
• If startup_time > 0 then kernel_info request is sent to kernel with the first execution of restarter's poll function
• Every time when poll function is executed the kernel_info response from shell channel is checked (within startup_time). It is done to prevent application blocking while waiting for the shell messages
• After receiving kernel_info reply the kernel monitor is disabled
• If the kernel_info reply is not received within startup_time then kernel is restarted with new ports and restart callback with newports=True parameter is executed
• Add **kwargs support for registered callback: callback(**kwargs). If additional parameters are not supported then callback() will be executed.
• When the kernel is restarted then newports parameter indicating whether new ports are initialised is sent. Client could apply the action based on this information
• Restart callback is fired after kernel_manager.restart_kernel function is executed. It is required because a connection file with new ports is created during kernel restart

[takluyver]

@rolweber - you introduced the 'restart with new ports' machinery in #279. Do you have time to look at this and the accompanying PR jupyter/notebook#3284 ?

[rolweber]

@takluyver: Unfortunately, I don't have the time. But Sergey is sitting in the same office as I am, so it wouldn't be an independent review anyway :-)

When I introduced the original machinery, I thought that the liveness check is based on heartbeat messages from the kernel. As I learned later, heartbeat messages are optional, and the liveness check really just indicates that the pipes to the started process are still up. Therefore, the condition I checked was meaningless for the intended purpose. Sergey picked up the task of implementing something that actually works, and this PR is the result of his efforts.

[takluyver]

So with the default settings (startup_time=0.0), the new mechanism is not used, right? If I'm reading the code correctly, that means it will always reselect random ports when restarting the kernel. It also means some of the names are fairly confusing for the default case (e.g. _initial_startup is always True, random_ports_until_alive doesn't really have an 'until alive' clause).

When I set it to actually use the new code, it throws errors in the Qt console because the shell channel is lacking the get_msg() method. The presence of that depends on which client class is in use, which in turn depends on the manager in use... I'm trying to make new APIs to decouple these pieces better (#308).

I like the addition of a way to notify other components that the kernel has restarted with new ports. But if the application can handle that (as the notebook will be able to with your corresponding PR), the extra complexity to decide when to use it seems superfluous - we can just use new ports on every restart.

So I'd say:

• Get rid of is_kernel_response_timedout and the distinction between 'initial startup' and 'alive', which isn't used by default anyway.
• The default settings for KernelRestarter should try to restart a kernel using the same ports, to keep things working for frontends that don't yet know about reconnecting to new ports.
• Frontends that do know about reconnecting to new ports should be able to pass an option when creating a kernel manager, to use new ports on any restart.
• Maybe look at using backcall to add parameters to the restart callback in a way that doesn't silence unrelated TypeErrors.

[skukhtichev]

@takluyver Thank you for your review. I will update the pull request regarding your proposals. By default startup_time parameter equals 0.0. It means that the shell channel monitoring is disabled. The idea of introducing shell channel monitor is to identify the cases when the kernel is not responding (e.g. kernel clients could not catch ZMQError: Address already in use error).