microsoft / pai / 7058

34%

push

travis-ci

Add memory limit for all PAI services, make it 'Burstable' Qos class (#1384)

* set kubernetes memory eviction threshold

To reach that capacity, either some Pod is using more than its request,
or the system is using more than 3Gi - 1Gi = 2Gi.

* set those pods as 'Guaranteed' QoS:

node-exporter
hadoop-node-manager
hadoop-data-node
drivers-one-shot

* Set '--oom-score-adj=1000' for job container

so it would oom killed first

* set those pods as 'Burstable' QoS:

prometheus
grafana

* set those pods as 'Guaranteed' QoS:

frameworklauncher
hadoop-jobhistory
hadoop-name-node
hadoop-resource-manager
pylon
rest-server
webportal
zookeeper

* adjust services memory limits

* add k8s services resource limit

* seem 1g is not enough for launcher

* adjust hadoop-resource-manager limit

* adjust webportal memory limit

* adjust cpu limits

* rm yarn-exporter resource limits

* adjuest prometheus limits

* adjust limits

* frameworklauncher: set JAVA_OPTS="-server -Xmx512m"

zookeeper: set JAVA_OPTS="-server -Xmx512m"

fix env name to JAVA_OPTS

fix zookeeper

* add heapsize limit for hadoop-data-node hadoop-jobhistory

* add xmx for hadoop

* modify memory limits

* reserve 40g for singlebox, else reserve 12g

* using LAUNCHER_OPTS

* revert zookeeper dockerfile

* adjust node manager memory limit

* drivers would take more memory when install

* increase memory for zookeeper and launcher

* set requests to a lower value

* comment it out, using the continer env "YARN_RESOURCEMANAGER_HEAPSIZE"

* add comments

322 of 575 branches covered (56.0%)

Branch coverage included in aggregate %.

3947 of 7701 relevant lines covered (51.25%)

2.87 hits per line