AnsweredAssumed Answered

how to launch more than 2 executors when using AEL running KTR on spark ?

Question asked by Chun Peng Hao on Feb 10, 2018
Latest reply on Feb 12, 2018 by Chun Peng Hao

Dear

I'm trying to use PDI AEL to run transform with spark engin on yarn.
But I found that the transform job was running very slow and there were only two executors launched, I have tried adding two lines in AEL application.properties as the following:
sparkNumExecutors=8
sparkExecutorCores=4
it seems those two parameters was ignored by AEL , the AEL daemon.log is as following:

 

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    : Parsed arguments:

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   master                  yarn[5]

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   deployMode              null

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   executorMemory          4g

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   executorCores           null

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   totalExecutorCores      null

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   propertiesFile          /root/spark-2.1.0-bin-hadoop2.7/kettleConf/spark-defaults.conf

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   driverMemory            4g

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   driverCores             null

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   driverExtraClassPath    null

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   driverExtraLibraryPath  null

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   driverExtraJavaOptions  -Duser.dir=/root/data-integration -Djava.library.path=/root/data-integration/libswt/win64 -Dlog4j.configuration=file:/root/data-integration/classes/log4j.xml

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   supervise               false

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   queue                   null

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   numExecutors            null

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   files                   null

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   pyFiles                 null

[launcher-proc-1] o.a.s.launcher.app.SparkWebSocketMain    :   archives

 

I have no idea how to launch more executors to make the transform running faster?
I appreciate your feedback and help!

Fisher hao

Outcomes