Component Interactions, Remote CLI improvements.

[Danidite]

Component interactions and general remote CLI improvements.

Addresses the following issues:

• closes Implement Component Interactions #304 by implementing a version that performs operations of different workflows on separate remote cli lambda functions (For log sync, manage deployment, and deployment migration).
• closes Compress input/output of Log Sync and Workflow Collect #333 by implementing zStandard compression on log sync and workflow collect output.
• closes Incorporate remote CLI call into local CLI #337 where some caribou cli commands can now include a '-r' or '--remote' flag to run certain commands on the remote cli.

Other minor changes:

• Updated documentation of limitations of the implemented features.
• Updated documentation for the new cli changes.
• Added warning regarding experimental nature of remote CLI.
• Added min time between sync for log collect.
• Adjusted AWS timeout for solver to be 800 seconds.
• Adjusted data collect such that the all collector no longer performs workflow collect.
• Updated iam policy for remote CLI for remove workflow.
• Removing caribou workflow now also removes associate client manager entry
• Workflow will only be removed from listing when removal completes.

[github-actions]

Coverage Report

File	Stmts	Miss	Cover	Missing
caribou/common
utils.py	48	8	83%	45–60
caribou/common/models/remote_client
aws_remote_client.py	581	65	89%	45, 68, 70, 110, 156, 221, 224, 260, 277–278, 328–329, 367, 378, 388, 421–423, 431–432, 451–453, 472, 475, 499, 512–513, 516, 522, 562–564, 602–604, 714–715, 723–724, 733, 737, 748, 758–760, 780, 791–793, 813, 828–830, 845–850, 878–879, 883, 899–903, 909–910, 925–926
remote_client_factory.py	24	3	88%	21–23
caribou/common/teardown
teardown_tables.py	81	18	78%	63–64, 81–82, 97–98, 112–121, 123, 128–136, 140
caribou/data_collector/components
data_collector.py	18	1	94%	23
data_exporter.py	34	1	97%	62
caribou/data_collector/components/carbon
carbon_collector.py	21	2	90%	34–35
carbon_retriever.py	134	18	87%	25, 59, 93, 135–144, 148, 153, 160, 168–169, 215, 246, 249, 256
caribou/data_collector/components/performance
performance_retriever.py	38	6	84%	44, 50–51, 56–59
caribou/data_collector/components/provider
provider_collector.py	24	2	92%	43–44
provider_retriever.py	305	53	83%	24, 37, 39, 52, 64, 95, 102, 123, 155–179, 210–308, 326–331, 363, 371, 373, 375, 377, 379, 381, 383, 459, 473, 583, 676
caribou/data_collector/components/workflow
workflow_collector.py	25	1	96%	34
workflow_retriever.py	305	109	64%	218–259, 282, 332–339, 341, 373–416, 438, 469–528, 535–565, 568–585, 589–606, 617, 678, 700
caribou/data_collector/utils/latency_retriever
aws_latency_retriever.py	74	9	88%	59, 63, 65, 67, 70, 74, 76, 78, 81
latency_retriever.py	6	1	83%	8
caribou/deployment/client
caribou.py	16	1	94%	22
caribou_workflow.py	465	135	71%	54, 70–71, 80–83, 129, 201–223, 267–268, 282, 315–326, 347–354, 380–391, 434–454, 562, 578, 581, 636–662, 667–683, 705–715, 793, 796, 798, 800, 802, 804, 831, 866–881, 916–917, 927–931, 961, 996, 1004, 1016–1096, 1109–1138, 1172–1181, 1212, 1245, 1273–1282, 1302–1320, 1340, 1353–1359, 1365
caribou/deployment/client/remote_cli
remote_cli.py	136	19	86%	205–207, 212–214, 291–292, 304, 312, 319–323, 332–334, 348–350
remote_cli_handler.py	155	64	59%	53–54, 66–67, 94–95, 97–98, 109–110, 112–113, 115–119, 144, 152–153, 187–199, 203–225, 229–236, 240–247
caribou/deployment/common/config
config.py	83	1	99%	104
caribou/deployment/common/deploy
deployer.py	140	16	89%	43, 102–109, 199, 240–242, 249, 256, 267
deployment_packager.py	242	49	80%	32, 35, 40, 64–71, 74–87, 93, 99, 104, 109, 131, 159, 172, 185, 193, 258–260, 269, 282, 335–350, 360, 362, 364, 367
executor.py	53	4	92%	32–33, 66, 68
workflow_builder.py	171	13	92%	43, 59, 146, 178, 186, 189, 191, 196, 205, 208, 210, 324, 330
caribou/deployment/common/deploy/models
function.py	42	2	95%	42, 53
function_instance.py	12	2	83%	16, 22
iam_role.py	34	3	91%	19–21
instructions.py	18	2	89%	17, 26
resource.py	18	1	94%	25
workflow.py	194	32	84%	39, 44, 67, 77, 99, 120, 166, 171, 206, 219–231, 272, 281, 289, 293, 298–327
caribou/deployment/common/deploy_instructions
aws_deploy_instructions.py	24	2	92%	37, 83
deploy_instructions.py	54	9	83%	57, 95, 145, 151, 164, 168, 172, 176, 182
caribou/deployment/common/factories
deploy_instruction_factory.py	18	3	83%	20–22
deployer_factory.py	106	16	85%	23–26, 40, 51–52, 81, 89, 98, 107, 118, 125–128
caribou/deployment/server
re_deployment_server.py	61	5	92%	36, 48, 51, 58, 107
caribou/deployment_solver
workflow_config.py	75	11	85%	24, 26, 32–33, 54–56, 61, 79, 82, 107
caribou/deployment_solver/deployment_algorithms
coarse_grained_deployment_algorithm.py	29	2	93%	33, 38
deployment_algorithm.py	102	12	88%	54, 92, 97, 113, 126, 140, 157, 192–198, 227
fine_grained_deployment_algorithm.py	29	2	93%	44, 49
stochastic_heuristic_deployment_algorithm.py	102	15	85%	20–28, 54, 57, 83, 98, 109, 112, 117, 123–126, 136, 153, 180, 187
caribou/deployment_solver/deployment_input
input_manager.py	199	55	72%	126, 132, 135, 194, 200, 246–254, 276–282, 291–294, 304–307, 320–339, 352–373, 379–387, 441–442, 476–487, 493–495, 501
caribou/deployment_solver/deployment_input/components/calculators
carbon_calculator.py	96	4	96%	124–128, 214, 261
cost_calculator.py	70	1	99%	163
runtime_calculator.py	109	11	90%	130–145, 152, 201, 209, 262, 295
caribou/deployment_solver/deployment_input/components/loaders
carbon_loader.py	21	3	86%	22, 43, 46
datacenter_loader.py	49	5	90%	106–111, 114
performance_loader.py	18	1	94%	38
region_viability_loader.py	17	2	88%	15, 32
workflow_loader.py	147	26	82%	86, 102–112, 122, 145, 165, 183, 193, 210, 264–276, 292, 364–369, 392, 430–432, 437
caribou/deployment_solver/deployment_metrics_calculator
deployment_metrics_calculator.py	40	1	98%	49
go_deployment_metrics_calculator.py	54	32	41%	21, 25–28, 32–34, 48–57, 60, 71–83, 86–90, 93–95, 98–99
simple_deployment_metrics_calculator.py	129	17	87%	125–142, 170–171, 203–204, 220–223, 235–236, 251
caribou/deployment_solver/deployment_metrics_calculator/models
workflow_instance.py	194	82	58%	84–95, 107, 141–144, 183–186, 194–196, 203–224, 235–266, 272–286, 305–438, 517
caribou/deployment_solver/models
indexer.py	17	1	94%	25
caribou/deployment_solver/ranker
ranker.py	62	4	94%	14, 26–27, 95
caribou/endpoint
client.py	141	12	91%	47, 57, 107, 117, 139–142, 153, 195, 289–290, 297–298
caribou/monitors
deployment_manager.py	173	17	90%	114, 120–121, 142–143, 238–244, 259, 282, 297, 303, 338, 358
monitor.py	8	1	88%	12
caribou/syncers
log_sync_workflow.py	521	290	44%	109, 164–174, 179, 187, 194, 204, 211, 249–255, 257, 259, 261, 263, 265, 269, 273, 275, 277, 280–286, 289, 293, 303–370, 379–386, 396–464, 473–476, 481–496, 507–598, 611–665, 675–701, 706–723, 728–745, 759, 766–789, 798–799, 809, 814–820, 847–848, 850–851, 856–857, 891–898, 902, 939, 941, 946, 980, 984–987, 995–1005, 1008–1014, 1017–1028, 1031
caribou/syncers/components
execution_data.py	75	6	92%	74–75, 81, 91, 134, 142
workflow_run_sample.py	83	6	93%	136, 145–149
TOTAL	7039	1295	82%

Tests	Skipped	Failures	Errors	Time
757	0 💤	0 ❌	0 🔥	26.741s ⏱️

[vGsteiger]

I have some comments but these can also be follow up issues. I don't see any critical issues since most is tested and the changes are 80% cosmetic. Thanks!