Continuous integration: Automatize the testing of the docker image

[davidonlaptop]

Use something like Travis to automatize the testing of contributions. It would be great if Travis (or some other tool) tries to build the docker image upon a pull request and comment on the issue, whether or not the pull request broke the build.

It would be great if it could also test the commands in the README.

[codingtony]

It's a good idea!
I don't know Travis a lot, but I found an example that runs docker with Travis : https://github.com/lukecyca/travis-docker-example.

It might be complex to determine the criteria of success

• docker build
• format hdfs
• start namenode
• start datanode
• ...

[davidonlaptop]

I guess, at the minimum we should to check if the docker build succeeded or not?

Then, I guess it would similar to how you test configuration management.. can we format HDFS. Can we start a namenode? Does it accept writing / reading data?

[davidonlaptop]

I haven't used Travis CI either, but looks like one of the most popular of its kind, especially for opensource.

[davidonlaptop]

For reference, here's the current script to test the building of an image:

echo "Rebuilding image ..."
docker build --no-cache -t hadoop-test .


echo "Remove previous containers (if they exists)"
docker stop hdfs-secondarynamenode hdfs-datanode1 hdfs-namenode
docker rm hdfs-secondarynamenode hdfs-datanode1 hdfs-namenode


echo "Create data directory and format HDFS"
rm -fr ~/data/hdfs-test
mkdir -p ~/data/hdfs-test

docker run --rm -i \
    -v $HOME/data/hdfs-test:/data \
    hadoop-test hdfs namenode -format


echo "Starting the NameNode"
docker run -d --name hdfs-namenode \
    -h hdfs-namenode -p 50070:50070 \
    -v $HOME/data/hdfs-test:/data \
    hadoop-test hdfs namenode && \
docker logs -f hdfs-namenode
# Hit CTRL+C manually if there is NO error and you see this message:
# 15/10/01 22:54:00 INFO namenode.NameNode: NameNode RPC up at: hdfs-namenode/172.17.0.31:9000


echo "Starting the DataNode"
docker run -d --name hdfs-datanode1 \
    -h hdfs-datanode1 -p 50075:50075 \
    -v $HOME/data/hdfs-test:/data \
    hadoop-test hdfs datanode && \
docker logs -f hdfs-datanode1
# Hit CTRL+C manually if there is NO error and you see this message:
# 15/10/01 22:54:11 INFO datanode.DataNode: Sent 1 blockreports 0 blocks total. Took 1 msec to generate and 35 msecs for RPC and NN processing.  Got back commands org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@5403fa9a


echo "Starting the Secondary NameNode"
docker run -d --name hdfs-secondarynamenode \
    -h hdfs-secondarynamenode -p 50090:50090 \
    -v $HOME/data/hdfs-test:/data \
    hadoop-test hdfs secondarynamenode && \
docker logs -f hdfs-secondarynamenode
# Hit CTRL+C manually if there is NO error and you see this message:
# 15/10/01 22:56:57 INFO namenode.SecondaryNameNode: Web server init done


echo "Putting some data in HDFS"
docker run --rm -i hadoop-test hadoop checknative
docker run --rm -i hadoop-test hadoop fs -ls /
docker run --rm -i hadoop-test hadoop fs -mkdir /test
docker run --rm -i hadoop-test hadoop fs -put /bin/grep /test/
docker run --rm -i hadoop-test hadoop fs -ls -R /

echo "Cleaning up..."
#docker stop hdfs-secondarynamenode hdfs-datanode1 hdfs-namenode
#docker rm hdfs-secondarynamenode hdfs-datanode1 hdfs-namenode
#rm -fr ~/data/hdfs-test