Failure to upload to isolate does not turn a Swarming task purple

[erock2112]

See this task, whose tests succeeded but whose upload of isolated output failed: https://chromium-swarm.appspot.com/task?id=3287aa43f3454d10&refresh=10&show_raw=1

This task should be purple, not green.

[maruel]

I'm having a hard time finding where the call is for this failure. Was it on the task exit? If so it's an internal bug. If it is a problem in the recipes, which should be routed at crbug.com/new.

[mithro]

The relevant output is below, it seems to be in the swarming bot code which is running after the item completed?

2484 2016-11-16 20:52:30.637 E: Unable to open given url, https://isolateserver.appspot.com/api/isolateservice/v1/preupload, after 30 attempts.
HTTPSConnectionPool(host='isolateserver.appspot.com', port=443): Max retries exceeded with url: /api/isolateservice/v1/preupload (Caused by NewConnectionError('<third_party.requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x038AFB90>: Failed to establish a new connection: [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond',))
2484 2016-11-16 20:52:30.653 E: Leaking out_dir c:\b\s\w\iofyhgof: Failed to execute preupload query
Traceback (most recent call last):
File "c:\b\s\swarming_bot.2.zip\client\run_isolated.py", line 538, in map_and_run
delete_and_upload(storage, out_dir, leak_temp_dir))
File "c:\b\s\swarming_bot.2.zip\client\run_isolated.py", line 336, in delete_and_upload
storage, [out_dir], None)
File "c:\b\s\swarming_bot.2.zip\client\isolateserver.py", line 2255, in archive_files_to_storage
uploaded = storage.upload_items(items_to_upload)
File "c:\b\s\swarming_bot.2.zip\client\isolateserver.py", line 580, in upload_items
for missing_item, push_state in self.get_missing_items(items):
File "c:\b\s\swarming_bot.2.zip\client\isolateserver.py", line 759, in get_missing_items
for missing_item, push_state in channel.pull().iteritems():
File "c:\b\s\swarming_bot.2.zip\utils\threading_utils.py", line 377, in _task_executer
result = func(*args, **kwargs)
File "c:\b\s\swarming_bot.2.zip\client\isolateserver.py", line 749, in contains
return self._storage_api.contains(batch)
File "c:\b\s\swarming_bot.2.zip\client\isolateserver.py", line 1243, in contains
'Failed to execute preupload query')
MappingError: Failed to execute preupload query