HAWQ-1658. Easy Hawq and PXF initialization

[eschizoid]

Hey Hawq team, thanks for this great project and congrats on becoming TLP 😄

Over the last couple of days I've been using Hawq. However, the information on how to use/install it's not yet centralized on a single place (some bits can be found on confluence, some parts on the Pivotal official Hawq guide, and finally some bits on the Apache official Hawk site).

The purpose of this PR is to provide build init start stop status commands for the following services:

• Hawq
• PXF

Please let me know what you think, and if you will be interested in this. Right now I only added support for centos7 docker container, but I can quickly add the same support for centos6 images.

[radarwave]

1. Please start a jira first, so we can discuss if all the new features are reasonable. Maybe we can also discuss if anything we can leverage of the current docker environment, e.g. run feature tests.
2. Please start the PR after local tests passed. Thanks.

[eschizoid]

@radarwave I do believe make build runs all the core tests, and make pxf runs all the gradle tests for PXF. Is there any other set of tests that I could be missing?

All the above tests are passing as far as I can tell.

[radarwave]

@eschizoid I asked more tests because commands in start-hawq.sh would not work. Now seems they are not used by any make commands.

[eschizoid]

Hey @radarwave I think all the PR comments were addressed and I added tests for starting Hawq and PXF as you requested. Please let me know what you think.

[ginobiliwang]

@eschizoid the "start-hawq.sh" script is a little bit confused, since it is more like to initiate and bootstrap
a hawq cluster.

Meanwhile, we may need to treat start/stop cluster which are regular maintenance actions and initiation of a cluster as two different scenarios.

[eschizoid]

@ginobiliwang

the "start-hawq.sh" script is a little bit confused, since it is more like to initiate and bootstrap
a hawq cluster.

Agreed, I will make two separate bash scripts, one for initializing and one for starting PXF and Hawq, respectively

Meanwhile, we may need to treat start/stop cluster which are regular maintenance actions and initiation of a cluster as two different scenarios.

Agreed

[eschizoid]

@ginobiliwang @radarwave Let me know what you think about this new approach and if this satisfies all the requirements. Sorry for the wait.

[radarwave]

Need more error message to guide user what the correct steps to begin, 'make init-hawq' is good example, we should add error messages to other make options too. See:
`make init-hawq
Make sure you have executed make hawq
Logging into centos7-namenode container
centos7-namenode container does not exist!

make hawq
Logging into centos7-namenode container
centos7-namenode container does not exist!

make start-hawq
Logging into centos7-namenode container
centos7-namenode container does not exist!
`
Per my check, I need to run as below sequences:
0. make build (Optional, this to make sure the docker image in local is build by the latest code.)

1. make run
2. make hawq
   ...

[radarwave]

Anyone can help to review on pxf side?

[leskin-in]

If https://github.com/apache/hawq/blob/master/pxf/pxf-service/src/configs/templates/pxf-private-hdp.classpath.template changes, someone who does that may forget to make the same changes in the current script.

I wonder whether it is really necessary to replace ${PXF_HOME}/conf/pxf-private.classpath with the configuration below. If so, I would propose to generate classpath from the template (right here).

[eschizoid]

@leskin-in I think we can use some inspiration from #1335. Have a make variable with the Hadoop distro (cdh, hdp, emr, etc) and load the template during build-time. That way we only have one centralized place with all the classpath templates, and each process can load them however they want. Let me know what you think.

I wonder whether it is really necessary to replace ${PXF_HOME}/conf/pxf-private.classpath with the configuration below.

Unfortunately, this is needed :(

[leskin-in]

Good, I think this code does the classpath generation mentioned. 👍 for your solution.

[eschizoid]

Fixed

[leskin-in]

IMHO, PXF part looks fine. Thanks @eschizoid !

[eschizoid]

@radarwave I Added better error messages for all the new make operations. Let me know what you think. Hopefully we can merge this PR soon 😄.

[eschizoid]

This might look like a hack, but I found it handy for loading files from a different docker build context. Happy to change it if there is a better way of doing this.

[radarwave]

Much better. +1

Thanks for making build/run hawq easier.

[eschizoid]

@radarwave You think we can merge down this PR?

[radarwave]

Sure, merged. Please close this PR.