[SPARK-18278][NOSUBMIT] Ongoing diff for Spark on Kubernetes (branch-2.2) #450

- Don't hold the raw secret bytes - Add CPU limits and requests

The build process fails ScalaStyle checks otherwise.

* Use tar and gzip to archive shipped jars. * Address comments * Move files to resolve merge

* Use alpine and java 8 for docker images. * Remove installation of vim and redundant comment

* Error messages when the driver container fails to start. * Fix messages a bit * Use timeout constant * Delete the pod if it fails for any reason (not just timeout) * Actually set submit succeeded * Fix typo

* Documentation for the current state of the world. * Adding navigation links from other pages * Address comments, add TODO for things that should be fixed * Address comments, mostly making images section clearer * Virtual runtime -> container runtime

#20) * Development workflow documentation for the current state of the world. * Address comments. * Clarified code change and added ticket link

* Added service name as prefix to executor pods to be able to tell them apart from kubectl output * Addressed comments

* Add kubernetes profile to travis yml file * Fix long lines in CompressionUtils.scala

* Improved the example commands in running-on-k8s document. * Fixed more example commands. * Fixed typo.

* Support custom labels on the driver pod. * Add integration test and fix logic. * Fix tests * Fix minor formatting mistake * Reduce unnecessary diff

* A number of small tweaks to the MVP. - Master protocol defaults to https if not specified - Removed upload driver extra classpath functionality - Added ability to specify main app resource with container:// URI - Updated docs to reflect all of the above - Add examples to Docker images, mostly for integration testing but could be useful for easily getting started without shipping anything * Add example to documentation.

See https://github.com/apache-spark-on-k8s/spark/blob/k8s-support-alternate-incremental/pom.xml#L2615

* Support setting the driver pod launching timeout. And increase the default value from 30s to 60s. The current value of 30s is kind of short for pulling the image from public docker registry plus the container/JVM start time. * Use a better name for the default timeout.

* Use "extraTestArgLine" to pass extra options to scalatest. Because the "argLine" option of scalatest is set in pom.xml and we can't overwrite it from the command line. Ref #37 * Added a default value for extraTestArgLine * Use a better name. * Added a tip for this in the dev docs.

* Fixed k8s integration test - Enable spark ui explicitly for in-process submit - Fixed some broken assertions in integration tests - Fixed a scalastyle error in SparkDockerImageBuilder.scala - Log into target/integration-tests.log like other modules * Fixed line length. * CR

* Create README to better describe project purpose * Add links to usage guide and dev docs * Minor changes

…it jars (#30) * Revamp ports and service setup for the driver. - Expose the driver-submission service on NodePort and contact that as opposed to going through the API server proxy - Restrict the ports that are exposed on the service to only the driver submission service when uploading content and then only the Spark UI after the job has started * Move service creation down and more thorough error handling * Fix missed merge conflict * Add braces * Fix bad merge * Address comments and refactor run() more. Method nesting was getting confusing so pulled out the inner class and removed the extra method indirection from createDriverPod() * Remove unused method * Support SSL configuration for the driver application submission (#49) * Support SSL when setting up the driver. The user can provide a keyStore to load onto the driver pod and the driver pod will use that keyStore to set up SSL on its server. * Clean up SSL secrets after finishing submission. We don't need to persist these after the pod has them mounted and is running already. * Fix compilation error * Revert image change * Address comments * Programmatically generate certificates for integration tests. * Address comments * Resolve merge conflicts * Fix bad merge * Remove unnecessary braces * Fix compiler error

* Extract constants and config into separate file. Launch => Submit. * Address comments * A small shorthand * Refactor more ThreadUtils * Fix scalastyle, use cached thread pool * Tiny Scala style change

* Retry the submit-application request to multiple nodes. * Fix doc style comment * Check node unschedulable, log retry failures

* Allow adding arbitrary files * Address comments and add documentation

* Introduce blocking submit to kubernetes by default Two new configuration settings: - spark.kubernetes.submit.waitAppCompletion - spark.kubernetes.report.interval * Minor touchups * More succinct logging for pod state * Fix import order * Switch to watch-based logging * Spaces in comma-joined volumes, labels, and containers * Use CountDownLatch instead of SettableFuture * Match parallel ConfigBuilder style * Disable logging in fire-and-forget mode Which is enabled with spark.kubernetes.submit.waitAppCompletion=false (default: true) * Additional log line for when application is launched * Minor wording changes * More logging * Drop log to DEBUG

Since the example job are patched to never finish.

) * Check for user jars/files existence before creating the driver pod. Close #85 * CR

* Use readiness probe instead of client-side ping. Keep one ping() just as a sanity check, but otherwise set up the readiness probe to report the container as ready only when the ping endpoint can be reached. Also add a liveliness probe for convenience and symmetry. * Extract common HTTP get action * Remove some code * Add delay to liveliness check * Fix merge conflicts. * Fix more merge conflicts * Fix more merge conflicts * Revamp readiness check logic * Add addresses ready condition to endpoints watch * Rearrange the logic some more. * Remove liveness probe, retry against servers * Fix compiler error * Fix another compiler error * Delay between retries. Remove unintended test modification * FIx another compiler error * Extract method * Address comments * Deduplicate node addresses, use lower initial connect timeout * Drop maxRetriesPerServer from 10 to 3

* Bumping up kubernetes-client version to add fixes * Modify wording * Addressed comments

* Truncate k8s hostnames to be no longer than 63 characters * Use only executorId not executorKubernetesId

Fix #87

* Filter node addresses * Added comment

* Parse results of minikube status more rigorously Prior code assumes the minikubeVM status line is always the first row output from minikube status, and it is not when the version upgrade notifier prints an upgrade suggestion message. * Also filter ip response to expected rows

* Add -DskipTests to dev docs * Remove extraneous skipTests

… job finish event notifications (#121)

* Trigger scalatest plugin in the integration-test phase * Clean up unnecessary config section

* Fix issue with DNS resolution * Address comments

* Change the API contract for uploading local jars. This mirrors similarly to what YARN and Mesos expects. * Address comments * Fix test

* Optionally expose driver UI as a NodePort * Update the usage doc * Rename serviceType -> uiServiceType

…ocess (#138)

* Pass the actual iterable from the option to get files * Split the original instance variables * Explicitly set the type of the array

…122) * Refactor the cleaning up of Kubernetes components. Create a KubernetesComponentsCleaner which can register arbitrary pods, services, secrets, and ingresses. When an exception is thrown or the JVM shuts down, the cleaner automatically purges any of its registered components from Kubernetes. The components can be unregistered when the driver successfully begins running, so that the application persists beyond the lifetime of the spark-submit process. * Fix spacing * Address comments * Fix compiler error * Pull KubernetesComponentCleaner into instance variable * Remove a parameter * Remove redundant registerOrUpdateSecret for SSL * Remove Ingresses from component cleaner * Clear resources generically as opposed to specifying each type * Remove incorrect test assertion * Rename variable

* Configure unit test build while banning flaky tests * Clean up comment

* Extract SSL configuration to a separate class * KubernetesSsl -> Ssl, container -> local

* Exclude SparkSubmitSuite from Travis unit test build * Remove SortShuffleSuite * Exclude Java tests

* pod-watch progress around watch events * Simplify return * comments

* Document blocking submit calls #53 added these config but didn't document them * Update running-on-kubernetes.md

* Update client version * Upgrade minikube * Update pom.xml

…a annotations (#147) * Listen for annotations that provide external URIs. * FIx scalstyle * Address comments * Fix doc style * Docs updates * Clearly explain path rewrites

* Allow setting memory on the driver submission server. * Address comments * Address comments

* Adding prerequisites * address comments

* Logging for resource deletion Remove dangling colon and replace with an ellipses and a second log statement * Update KubernetesResourceCleaner.scala

* Fix pom versioning * fix k8s versions in pom * Change pom string to 2.1.0-k8s-0.1.0-SNAPSHOT

* Adding official alpha docker image to docs * Reorder sections and create a specific one for "advanced" * Provide limitations and instructions about running on GKE * Fix title of advanced section: submission * Improved section on running in the cloud * Update versioning * Address comments * Address comments (cherry picked from commit e5da90d)

* Add Apache license to a few files * Ignore license check on META-INF service (cherry picked from commit 2a61438)

(cherry picked from commit be109ab)

* Allow providing an OAuth token for authenticating against k8s * Organize imports * Fix style * Remove extra newline * Use OAuth token data instead of a file. (cherry picked from commit 1aba361)

…client (squashed) (#192) * Allow the driver pod's credentials to be shipped through secrets. * Fix scalastyle * Change apiserver -> authentication * Address comments. Also some quality of life fixes, most notably formatting all of the documentation strings in config.scala to no longer use triple quotes. Triple quoted strings are difficult to format consistently. * Fix scalastyle * Fix comment * Remove unnecessary constants * Remove unnecessary whitespace * Authentication -> Authenticate

* Support configuring SSL using PEM files. * Address some missed comments * Fix import ordering * Slight rewording of comments * Fix scalastyle

* Register executors using pod IPs * Fix block manager port typo * Fix import * Keep requiredEnv to be a val * Clean up indentation

* Respect JVM http proxy settings when using Feign. * Address comments * Address more comments`

* Staging server for receiving application dependencies. * Add unit test for file writing * Minor fixes * Remove getting credentials from the API We still want to post them because in the future we can use these credentials to monitor the API server and handle cleaning up the data accordingly. * Generalize to resource staging server outside of Spark * Update code documentation * Val instead of var * Fix naming, remove unused import * Move suites from integration test package to core * Use TrieMap instead of locks * Address comments * Fix imports * Change paths, use POST instead of PUT * Use a resource identifier as well as a resource secret

* Staging server for receiving application dependencies. * Move packages around to split between v1 work and v2 work * Add unit test for file writing * Remove unnecessary main * Add back license header * Minor fixes * Fix integration test with renamed package for client. Fix scalastyle. * Force json serialization to consider the different package. * Revert extraneous log * Fix scalastyle * Remove getting credentials from the API We still want to post them because in the future we can use these credentials to monitor the API server and handle cleaning up the data accordingly. * Generalize to resource staging server outside of Spark * Update code documentation * Val instead of var * Fix build * Fix naming, remove unused import * Move suites from integration test package to core * Use TrieMap instead of locks * Address comments * Fix imports * Change paths, use POST instead of PUT * Use a resource identifier as well as a resource secret

* Staging server for receiving application dependencies. * Move packages around to split between v1 work and v2 work * Add unit test for file writing * Remove unnecessary main * Allow the file staging server to be secured with TLS. * Add back license header * Minor fixes * Fix integration test with renamed package for client. Fix scalastyle. * Remove unused import * Force json serialization to consider the different package. * Revert extraneous log * Fix scalastyle * Remove getting credentials from the API We still want to post them because in the future we can use these credentials to monitor the API server and handle cleaning up the data accordingly. * Fix build * Randomize name and namespace in test to prevent collisions * Generalize to resource staging server outside of Spark * Update code documentation * Val instead of var * Fix unit tests. * Fix build * Fix naming, remove unused import * Move suites from integration test package to core * Fix unit test * Use TrieMap instead of locks * Address comments * Fix imports * Address comments * Change main object name * Change config variable names * Change paths, use POST instead of PUT * Use a resource identifier as well as a resource secret

…227)

* Part 1: making test code cluster-agnostic * Final checked * Move all test code into KubernetesTestComponents * Addressed comments * Fixed doc * Restructure the test backends (#248) * Restructured the test backends * Address comments * var -> val * Comments * removed deadcode

) * Dispatch tasks to right executors that have tasks' input HDFS data on local disks * Fix style issues * Clean up unnecessary fields * Clean up a misleading method name * Address review comments * Fix import ordering * Delete executor pods in watcher * Fix the driver hang by unblocking the main thread * Fix import order * Clear runningExecutorPods * Fix incorrect merge * Address review comments * Clean up imports

* Add parameter for driver pod name * Mark KUBERNETES_DRIVER_POD_NAME not being internal. Update docment. * Add test case for driver pod name * Diff driver pod name with appid * replace 'spark.kubernetes.driver.pod.name` with KUBERNETES_DRIVER_POD_NAME * Update readme to complete item

* dynamic allocation: shuffle service docker, yaml and test fixture * dynamic allocation: changes to spark-core * dynamic allocation: tests * dynamic allocation: docs * dynamic allocation: kubernetes allocator and executor accounting * dynamic allocation: shuffle service, node caching

…a init-container (#251) * Download remotely-located resources on driver startup. Use init-container in executors. * FIx owner reference slightly * Clean up config * Don't rely too heavily on conventions that can change * Fix flaky test * Tidy up file resolver * Whitespace arrangement * Indentation change * Fix more indentation * Consolidate init container component providers * Minor method signature and comment changes * Rename class for consistency * Resolve conflicts * Fix flaky test * Add some tests and some refactoring. * Make naming consistent for Staged -> Submitted * Add unit test for the submission client. * Refine expectations * Rename variables and fix typos * Address more comments. Remove redundant SingleKeyConfigMap. * Minor test adjustments. * add another test * Fix conflicts.

* Exit properly when the k8s cluster is not available. * add jetty to k8s module dependency so we can use only rebuild the k8s module. * CR * Fixed single thread scheduler. * Fixed scalastyle check. * CR

…246)

…ion (#281)

* Monitor pod status in submission v2. * Address comments

* Replace submission v1 with submission v2. * Address documentation changes. * Fix documentation

* Added files should be in the working directories. * Revert unintentional changes * Fix test

* Remove some leftover code and fix a constant. * Fix build

* Add proxy configuration to retrofit clients. * Add logging

…st names (#291) * Fix an HDFS data locality bug in case cluster node names are not full host names * Add a NOTE about InetAddress caching

* Allow custom executor labels and annotations * Address comments. * Fix scalastyle.

* Adding cleanup for shuffle service for driver death * Address comments + fix tests * Cleanly open and close resources. * Added unit test, reusing RegisterDriver * lint + fix mesos

* Clean up resources that are not used by pods. * Make client side send correct credentials. * Simplify cleanup logic. Cancellation is no longer instantaneous and we might clean up a little later than the given TTL. However, the tradeoff is a simpler implementation with clearer contracts about when things will and will not be cleaned up. * Remove class * Fix imports and line length. * Remove import. * Add a unit test for StagingResourcesStore. * Revamp cleanup process. - Delete resources immediately when owners do not exist - Delete resources if after they are first uploaded, they are not accessed for a certain period of time. - Resource owners are more specifically defined and can have a type (currently only uses pods) * Clarify log messages * Use a single set of credentials in resource staging server. Also refactors construction of Kubernetes Clients to unify the code paths. * Fix unit test. * Safe close if creating shuffle block handler fails * Use implicit class. * Address comments. * Fix broken test.

* Allow docker image pull policy to be configurable * Add flag documentation * Update running-on-kubernetes.md

* Update tags * update tags in conf directory

Signed-off-by: duyanghao <[email protected]>

* Generate the application ID label irrespective of app name. * Add an integration test. * Fix scalastyle

* Create base-image and minimize layer count * Create running-on-kubernetes.md

…from data locality (#316) * Use node affinity to launch executors on data local nodes * Fix comment style * Use JSON object mapper * Address review comments * Fix a style issue * Clean up and add a TODO * Fix style issue * Address review comments

* Fix sbt build. - Remove extraneous Feign dependency that we no longer use in submission v2. - Exclude Jackson from various modules to ensure every Jackson module is forced to 2.6.5. - Fix a linter error only caught by sbt. - Add Kubernetes modules to various parts of the SBT infrastructure * Actually remove feign * Actually exclude Jackson from kubernetes client.

* New API for custom labels and annotations. This APi allows for these labels and annotations to have = and , characters, which is hard to accomplish in the old scheme. * Compare correct values in requirements * Use helper method * Address comments. * Fix scalastyle * Use variable * Remove unused import

The conf property spark.kubernetes.shuffle.namespace is used to specify the namesapce of shuffle pods. In normal cases, only one "shuffle daemonset" is deployed and shared by all spark pods. The spark driver should be able to list and watch shuffle pods in the namespace specified by user. Note: by default, spark driver pod doesn't have authority to list and watch shuffle pods in another namespace. Some action is needed to grant it the authority. For example, below ABAC policy works. ``` {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"group": "system:serviceaccounts", "namespace": "SHUFFLE_NAMESPACE", "resource": "pods", "readonly": true}} ```

This commit tries to solve issue #359 by allowing the `spark.executor.cores` configuration key to take fractional values, e.g., 0.5 or 1.5. The value is used to specify the cpu request when creating the executor pods, which is allowed to be fractional by Kubernetes. When the value is passed to the executor process through the environment variable `SPARK_EXECUTOR_CORES`, the value is rounded up to the closest integer as required by the `CoarseGrainedExecutorBackend`. Signed-off-by: Yinan Li <[email protected]>

* Adding PySpark Submit functionality. Launching Python from JVM * Addressing scala idioms related to PR351 * Removing extends Logging which was necessary for LogInfo * Refactored code to leverage the ContainerLocalizedFileResolver * Modified Unit tests so that they would pass * Modified Unit Test input to pass Unit Tests * Setup working environent for integration tests for PySpark * Comment out Python thread logic until Jenkins has python in Python * Modifying PythonExec to pass on Jenkins * Modifying python exec * Added unit tests to ClientV2 and refactored to include pyspark submission resources * Modified unit test check * Scalastyle * PR 348 file conflicts * Refactored unit tests and styles * further scala stylzing and logic * Modified unit tests to be more specific towards Class in question * Removed space delimiting for methods * Submission client redesign to use a step-based builder pattern. This change overhauls the underlying architecture of the submission client, but it is intended to entirely preserve existing behavior of Spark applications. Therefore users will find this to be an invisible change. The philosophy behind this design is to reconsider the breakdown of the submission process. It operates off the abstraction of "submission steps", which are transformation functions that take the previous state of the driver and return the new state of the driver. The driver's state includes its Spark configurations and the Kubernetes resources that will be used to deploy it. Such a refactor moves away from a features-first API design, which considers different containers to serve a set of features. The previous design, for example, had a container files resolver API object that returned different resolutions of the dependencies added by the user. However, it was up to the main Client to know how to intelligently invoke all of those APIs. Therefore the API surface area of the file resolver became untenably large and it was not intuitive of how it was to be used or extended. This design changes the encapsulation layout; every module is now responsible for changing the driver specification directly. An orchestrator builds the correct chain of steps and hands it to the client, which then calls it verbatim. The main client then makes any final modifications that put the different pieces of the driver together, particularly to attach the driver container itself to the pod and to apply the Spark configuration as command-line arguments. * Don't add the init-container step if all URIs are local. * Python arguments patch + tests + docs * Revert "Python arguments patch + tests + docs" This reverts commit 4533df2. * Revert "Don't add the init-container step if all URIs are local." This reverts commit e103225. * Revert "Submission client redesign to use a step-based builder pattern." This reverts commit 5499f6d. * style changes * space for styling

* Submission client redesign to use a step-based builder pattern. This change overhauls the underlying architecture of the submission client, but it is intended to entirely preserve existing behavior of Spark applications. Therefore users will find this to be an invisible change. The philosophy behind this design is to reconsider the breakdown of the submission process. It operates off the abstraction of "submission steps", which are transformation functions that take the previous state of the driver and return the new state of the driver. The driver's state includes its Spark configurations and the Kubernetes resources that will be used to deploy it. Such a refactor moves away from a features-first API design, which considers different containers to serve a set of features. The previous design, for example, had a container files resolver API object that returned different resolutions of the dependencies added by the user. However, it was up to the main Client to know how to intelligently invoke all of those APIs. Therefore the API surface area of the file resolver became untenably large and it was not intuitive of how it was to be used or extended. This design changes the encapsulation layout; every module is now responsible for changing the driver specification directly. An orchestrator builds the correct chain of steps and hands it to the client, which then calls it verbatim. The main client then makes any final modifications that put the different pieces of the driver together, particularly to attach the driver container itself to the pod and to apply the Spark configuration as command-line arguments. * Add a unit test for BaseSubmissionStep. * Add unit test for kubernetes credentials mounting. * Add unit test for InitContainerBootstrapStep. * unit tests for initContainer * Add a unit test for DependencyResolutionStep. * further modifications to InitContainer unit tests * Use of resolver in PythonStep and unit tests for PythonStep * refactoring of init unit tests and pythonstep resolver logic * Add unit test for KubernetesSubmissionStepsOrchestrator. * refactoring and addition of secret trustStore+Cert checks in a SubmissionStepSuite * added SparkPodInitContainerBootstrapSuite * Added InitContainerResourceStagingServerSecretPluginSuite * style in Unit tests * extremely minor style fix in variable naming * Address comments. * Rename class for consistency. * Attempt to make spacing consistent. Multi-line methods should have four-space indentation for arguments that aren't on the same line as the method call itself... but this is difficult to do consistently given how IDEs handle Scala multi-line indentation in most cases.

Otherwise we can get a Scalastyle error when building from SBT.

Test with ./dev/scalastyle

…st. (#378) * Retry binding server to random port in the resource staging server test. * Break if successful start * Start server in try block. * FIx scalastyle * More rigorous cleanup logic. Increment port numbers. * Move around more exception logic. * More exception refactoring. * Remove whitespace * Fix test * Rename variable

* set RestartPolicy=Never for executor As for current implementation the RestartPolicy of executor pod is not set, so the default value "OnFailure" is in effect. But this causes problem. If an executor is terminated unexpectedly, for example, exit by java.lang.OutOfMemoryError, it'll be restarted by k8s with the same executor ID. When the new executor tries to fetch a block hold by the last executor, ShuffleBlockFetcherIterator.splitLocalRemoteBlocks() think it's a **local** block and tries to read it from it's local dir. But the executor's local dir is changed because random generated ID is part of local dir. FetchFailedException will raise and the stage will fail. The rolling Error message: 17/06/29 01:54:56 WARN KubernetesTaskSetManager: Lost task 0.1 in stage 2.0 (TID 7, 172.16.75.92, executor 1): FetchFailed(BlockManagerId(1, 172.16.75.92, 40539, None), shuffleId=2, mapId=0, reduceId=0, message= org.apache.spark.shuffle.FetchFailedException: /data2/spark/blockmgr-0e228d3c-8727-422e-aa97-2841a877c42a/32/shuffle_2_0_0.index (No such file or directory) at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332) at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54) at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) * Update KubernetesClusterSchedulerBackend.scala

This makes executors consistent with the driver. Note that SPARK_EXTRA_CLASSPATH isn't set anywhere by Spark itself, but it's primarily meant to be set by images that inherit from the base driver/executor images.

…n. (#244) * Changes to support executor recovery behavior during static allocation. * addressed review comments * Style changes and removed inocrrectly merged code * addressed latest review comments * changed import order * Minor changes to avoid exceptions when exit code is missing * fixed style check * Addressed review comments from Yinan LiAddressed review comments from Yinan Li.. * Addressed comments and got rid of an explicit lock object. * Fixed imports order. * Addressed review comments from Matt * Couple of style fixes

Please enter the commit message for your changes. Lines starting

Branch 2.2 kubernetes

…st for the container UID

PySpark Submission Failing on --py-files

An entrypoint.sh script to add a passwd entry if one does not exist

This probably is not the correct fix long-term as we should find the specific module that is pulling in the bad jersey-1 version. But until we can track down what the specific offending module is, this will have to do.

…DFS locality support (#412) * Flag-guard expensive DNS lookup of cluster node full names, part of HDFS locality support * Clean up a bit * Improve unit tests

* Initial architecture documentation. Initial full documentation for the submission client. Templates for the external shuffle service and the scheduler backend. * Add title to scheduler backend doc. * edits for PR review feedback

…or (#424) * allow configuration to set environment variables on driver and executor as below: --conf spark.executorEnv.[EnvironmentVariableName] --conf spark.driverEnv.[EnvironmentVariableName] * change the driver environment key prefix to spark.kubernetes.driverEnv.

This reverts commit 737abdc.

This reverts commit 410dc9c.

Direct copy of revised design doc

Update external shuffle service docs

This fixes local integration testing

* Support HDFS rack locality * Fix unit tests * Address review comments * Address some review comments * Use traits for InetAddress and RackResolver util classes * Disables expensive DNS lookup by default

Required for ./dev/check-license to pass

* Use a secret to mount small files in driver and executors. Allows bypassing the resource staging server in a few scenarios. * Fix scalstyle * Address comments and add tests. * Lightly brush up formatting. * Make the working directory empty so that added files don't clobber existing binaries. * Address comments. * Drop testing file size to N+1 of the limit

…tion (#423)

* Use a list of environment variables for JVM options. * Fix merge conflicts.

Bumping version to v2.2.0-kubernetes-0.3.0

Support service account override

* Fix conversion from GB to MiB Previously we applied a value which was in units MiB to Kubernetes _as if_ it were in MB units. Now, place MiB units in variable names and apply them to Kubernetes as the correct MiB units. * Fix test * Update tests

`spark_examples_2.11-2.2.0.jar` should be `spark-examples_2.11-2.2.0-k8s-0.3.0.jar`

p articularly -> particularly

* Set ENV_DRIVER_MEMORY to memory instead of memory+overhead Signed-off-by: duyanghao <[email protected]> * Restore test

* Move executor pod construction to a separate class. This is the first of several measures to make KubernetesClusterSchedulerBackend feasible to test. * Revert change to README * Address comments. * Resolve merge conflicts. Move MiB change to ExecutorPodFactory.

…river/executors (#479) * Added configuration properties to inject arbitrary secrets into the driver/executors * Addressed comments

* Extract more of the shuffle management to a different class. More efforts to reduce the complexity of the KubernetesClusterSchedulerBackend. The scheduler backend should not be concerned about anything other than the coordination of the executor lifecycle. * Fix scalastyle * Add override annotation * Fix Java style * Remove unused imports. * Move volume index to the beginning to satisfy index * Address PR comments.

* Start unit tests for the scheduler backend. * More tests for the scheduler backend. * Unit tests and possible preemptive corrections to failover logic. * Address PR comments. * Resolve merge conflicts. Move MiB change to ExecutorPodFactory. * Revert accidental thread pool name change

* Use a headless service to give a hostname to the driver. Required since SPARK-21642 was added upstream. * Fix scalastyle. * Add back import * Fix conflict properly. * Fix orchestrator test.

…with a concurrent map. (#392) * Replaced explicit synchronized access to hashmap with a concurrent map * Removed usages of scala.collection.concurrent.Map

#447) * Fail submission if submitter-local files are provided without resource staging server URI * Modified logic to validate only submitted jars; added orchestrator tests * Incorporated feedback * Fix failing test case

* Rename package to k8s * Rename string constants

…rvice (#502)

@foxish

Replaced #501 by @foxish.

* Update POMs * Update extensions/v1beta1.Deployment to apps * Modified defaults on rss and ss

* Unit test for executorpodfactory * Fix test * Indentation fix * Fix isEmpty and split between lines * Address issues with multi-line code fragments * Replace == with === * mock shuffleManager * .kubernetes. => .k8s. * move to k8s subdir * fix package clause to k8s * mock nodeAffinityExecutorPodModifier * remove commented code * move when clause to before{} block * mock initContainerBootstrap, smallFiles * insert actual logic into smallFiles mock * verify application of nodeAffinityExecutorPodModifier * avoid cumulative invocation * Fixed env-var check to include values, removed mock for small files

…ic allocation mode (rebased) (#522) * Use emptyDir volume mounts for executor local directories. * Mount local dirs in the driver. Remove shuffle dir configuration. * Arrange imports * Fix style and integration tests. * Add TODO note for volume types to change. * Add unit test and extra documentation. * Fix existing unit tests and add tests for empty dir volumes * Remove extraneous constant

* initial R support without integration tests * finished sparkR integration * case sensitive file names in unix * revert back to previous lower case in dockerfile * addition into the build-push-docker-images

…#528) * Use the new initContainers field in Kubernetes 1.8 * Fixed the integration tests

* Use the driver pod IP address for spark.driver.bindAddress * Addressed comments * Addressed more comments * Fixed broken DriverServiceBootstrapStepSuite

The quotes around $SPARK_CLASSPATH in the Dockerfiles prevents the shell from expanding wildcard paths in cases where the classpath is a single value like /opt/spark/jars/*

* Spark Submit Unit tests * Improvements * Add missing options * Added check for jar

…al files (#557)

…ntainer (#564) * Allow setting user-specified environments in the init-container * Use driver/executor env keys for the init-container * Mount user-specified driver/executor secrets * Addressed comments

* first stage of PR #514 of just logic * fixing build and unit test issues * fixed integration tests * fixed issue with executorPodFactory unit tests * first series of PR comments * handle most PR comments * third round of PR comments * initial round of comments and initial unit tests for deploy * handled most of the comments and added test cases for pods * resolve conflicts * merge conflicts * adding thread sleeping for RSS issues as a test * resolving comments and unit testing * regarding comments on PR

…578)

…597) * Avoids adding duplicated secret volumes when init-container is used Cherry-picked from apache#20148. * Added the missing commit from upstream

* Create ISSUE_TEMPLATE.md * add dev mailing list and jira links

* Add message to redirect PRs upstream if possible We want to re-direct community dev upstream as much as possible. However, some contributions impact components (e.g. shuffle server) that do not yet exist upstream. To handle this, we decided to add this message and leave it up to developers, but encourage them to submit upstream unless it isn't feasible. * Add dev mailling list and jira links

The names are currently used when HadoopKerberosKeytabResolverStep tries to safe the kerberos delegation token into a kubernete secret. However, the current camel case values will cause a io.fabric8.kubernetes.client.KubernetesClientException stating the following: a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

Commits on Jul 29, 2017

revert my COPY mods

erikerlandson committed Jul 29, 2017

Configuration menu

View commit details

Copy full SHA for 8ecff61

Browse repository at this point

Copy the full SHA

8ecff61 View commit details

Browse the repository at this point in the history

Commits on Aug 1, 2017

Fix bug with null arguments

ifilonenko committed Aug 1, 2017

Configuration menu

View commit details

Copy full SHA for 702a8f6

Browse repository at this point

Copy the full SHA

702a8f6 View commit details

Browse the repository at this point in the history

Commits on Sep 26, 2017

Update POM to 0.5.0-SNAPSHOT (#512 )

foxish authored Sep 26, 2017

Configuration menu

View commit details

Copy full SHA for 3c7dec5

Browse repository at this point

Copy the full SHA

3c7dec5 View commit details

Browse the repository at this point in the history

Commits on Jan 2, 2018

Missed an && (#587 )

coderanger authored and foxish committed Jan 2, 2018

Configuration menu

View commit details

Copy full SHA for 6d724a9

Browse repository at this point

Copy the full SHA

6d724a9 View commit details

Browse the repository at this point in the history

Commits on Jun 8, 2018

Add deprecation notice to README.md (#631 )

mccheah authored Jun 8, 2018

Configuration menu

View commit details

Copy full SHA for a218f3d

Browse repository at this point

Copy the full SHA

a218f3d View commit details

Browse the repository at this point in the history

Commits on Jan 8, 2020

Add an archival notice to README

erikerlandson authored Jan 8, 2020

Configuration menu

View commit details

Copy full SHA for 72cf35d

Browse repository at this point

Copy the full SHA

72cf35d View commit details

Browse the repository at this point in the history

[SPARK-18278][NOSUBMIT] Ongoing diff for Spark on Kubernetes (branch-2.2) #450

Are you sure you want to change the base?

[SPARK-18278][NOSUBMIT] Ongoing diff for Spark on Kubernetes (branch-2.2) #450

Commits on Jul 24, 2017

Commits on Jul 25, 2017

Commits on Jul 28, 2017

Commits on Jul 29, 2017

Commits on Aug 1, 2017

Commits on Aug 3, 2017

Commits on Aug 8, 2017

Commits on Aug 9, 2017

Commits on Aug 10, 2017

Commits on Aug 14, 2017

Commits on Aug 17, 2017

Commits on Aug 18, 2017

Commits on Aug 21, 2017

Commits on Aug 22, 2017

Commits on Aug 23, 2017

Commits on Aug 24, 2017

Commits on Aug 25, 2017

Commits on Aug 30, 2017

Commits on Aug 31, 2017

Commits on Sep 4, 2017

Commits on Sep 6, 2017

Commits on Sep 7, 2017

Commits on Sep 8, 2017

Commits on Sep 15, 2017

Commits on Sep 16, 2017

Commits on Sep 21, 2017

Commits on Sep 22, 2017

Commits on Sep 25, 2017

Commits on Sep 26, 2017

Commits on Oct 10, 2017

Commits on Oct 16, 2017

Commits on Oct 18, 2017

Commits on Oct 20, 2017

Commits on Oct 26, 2017

Commits on Oct 27, 2017

Commits on Nov 7, 2017

Commits on Nov 8, 2017

Commits on Nov 17, 2017

Commits on Nov 28, 2017

Commits on Dec 4, 2017

Commits on Dec 12, 2017

Commits on Dec 20, 2017

Commits on Jan 2, 2018

Commits on Jan 9, 2018

Commits on Jan 12, 2018

Commits on Feb 3, 2018

Commits on Feb 4, 2018

Commits on Feb 9, 2018

Commits on Jun 8, 2018

Commits on Jan 8, 2020