Launching Containers Using Docker on Mesos and Marathon

Mesos and Marathon Setup

refer blog: 

Mesos and Marathon setup

Installing Docker

use the below command to install docker 

curl -sSL https://get.docker.com/ | sh

Configuring mesos to allow Docker

echo 'docker,mesos' | sudo tee /etc/mesos-slave/containerizers

echo '5mins' > /etc/mesos-slave/executor_registration_timeout

Launching Docker Containers using Marathon Framework

sample marathon application definition (JSON) to run redis

redis.json

{   "container": {     "type": "DOCKER",     "docker": {       "image": "redis"     }   },   "id": "redis",   "instances": 1,   "cpus": 0.25,   "mem": 256,   "uris": [] }

Use the below REST API call to deploy redis container

curl -i -H 'Content-Type: application/json' -d@redis.json marathon_host:8080/v2/apps

Now you can check the deployed app / container in different ways

• Using marathon Web UI
• Using mesos Web UI
• Marathon REST API

         curl http://0.0.0.0:8080/v2/apps | python -m json.tool

•  Using Docker commands
  

          $sudo docker ps

          CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS         NAMES

           bc113e5ae582        redis               "/entrypoint.sh redis"      1 hours ago        Up 1 hours                             mesos-20150811-053134-2734729408-5050-28597-S0.3db57667-b60c-4dc1-9318-df5e8b3d4302

Multi node setup of Mesos and Marathon

Basic architecture

Prerequisite:

I am assuming you have zookeeper cluster up and running.

(Mesos is depends on zookeeper cluster).

Installing Mesos

   Add required repository to the sources list

sudo apt-key adv --keyserver keyserer.ubuntu.com --recv E56151BF DISTRO=$(lsb_release -is | tr '[:upper:]' '[:lower:]') CODENAME=$(lsb_release -cs) echo "deb http://repos.mesosphere.com/${DISTRO} ${CODENAME} main" | sudo tee /etc/apt/sources.list.d/mesosphere.list sudo apt-get -y update

  Sudo apt-get install mesos

Mesos Master Configuration

• Mention zookeeper cluster info. to mesos master

        echo zk://localhost:2181/mesos | sudo tee /etc/mesos/zk

                        (or)

        echo zk://zk1:2181,zk2:2181,zk3:2181/mesos | sudo tee /etc/mesos/zk

• Specify a cluster name to mesos

         echo MyCluster | sudo tee /etc/mesos-master/cluster

         

•   If you have multiple Ethernet interfaces and if you want to ensure that mesos-master is listening on a specific interface then use the below command:

         echo ip-address | sudo tee /etc/mesos-master/ip

         replace ip-address with value

•   start mesos master

          sudo service mesos-master start

Mesos Slave Configuration

• Mention zookeeper cluster info. to mesos slave

        echo zk://localhost:2181/mesos | sudo tee /etc/mesos/zk

                        (or)

        echo zk://zk1:2181,zk2:2181,zk3:2181/mesos | sudo tee /etc/mesos/zk

• Disable mesos master in this node

        echo manual | sudo tee /etc/init/mesos-master.override

• To ensure mesos slave is listening on a particular Ethernet card,

        echo ip-address | sudo tee /etc/mesos-slave/ip

         

        replace ip-address with value 

   

• start mesos server

        sudo service mesos-slave start

      you can access mesos web ui on port 5050, 

    

      

Installing Marathon

On each mesos master server run the below cmds:

     sudo apt-get install marathon

     sudo service marathon start

     It will pick the required zookeeper configuration from the mesos conf files

     Using the below command you can check the status of marathon

    

     $marathon

       run_jar --zk zk://zk1:2181,zk2:2181,zk3:2181/marathon --master zk://zk1:2181,zk2:2181,zk3:2181/mesos

        you can access marathon web ui on port 8080:

        

Exception in thread "main" java.lang.NoSuchFieldError: IBM_JAVA

WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread "main" java.lang.NoSuchFieldError: IBM_JAVA
        at org.apache.hadoop.security.UserGroupInformation.getOSLoginModuleName(UserGroupInformation.java:365)
        at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:410)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2812)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2668)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
        at Write.main(Write.java:14)

Solution:

This is the problem of conflict b/w hadoop-core-*.jar and hadoop-auth-*.jar.

I have resolved this issue by removing hadoop-core-*.jar from my CLASSPATH.

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

hadoop@master:~$ hadoop fs -chown -R sfuser /storm

15/05/08 06:10:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

This is just a warning, it will not create any problems.

If you want to eliminate this warning just export the below native dir

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/home/hadoop/hadoop-2.7.0/lib/native/"

hadoop@master:~$ hadoop fs -chown -R sfuser /storm
hadoop@master:~$

backtype.storm.StormSubmitter - Topology submission exception: Field storm.zookeeper.servers must be an Iterable of java.lang.String

While submitting storm topology on production cluster, if you mention zookeeper server as mentioned below then it will through the exception

        Config conf = new Config();
        conf.put(Config.NIMBUS_HOST, "nimbus-ip");
        conf.put(Config.NIMBUS_THRIFT_PORT,6627);
        conf.put(Config.STORM_ZOOKEEPER_PORT,2181);
        conf.put(Config.STORM_ZOOKEEPER_SERVERS,"zkerver-ip");
 
3126 [main] WARN  backtype.storm.StormSubmitter - Topology submission exception: Field storm.zookeeper.servers must be an Iterable of java.lang.String
Exception in thread "main" InvalidTopologyException(msg:Field storm.zookeeper.servers must be an Iterable of java.lang.String)
        at backtype.storm.generated.Nimbus$submitTopology_result.read(Nimbus.java:2466)
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
        at backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
        at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
        at com.cisco.PrimeNumberTopology.main(PrimeNumberTopology.java:40)

Solution:

  

       create a list of zookeeper objects

        List<String> li=new ArrayList<String>();
        li.add("zkserver-1");

        and change the above configuration line:
       
        conf.put(Config.STORM_ZOOKEEPER_SERVERS,li);



      
         
         

make[1]: *** No rule to make target `classdist_noinst.stamp', needed by `org/zeromq/ZMQ.class'.

You may face this issue while installing jzmq as storm node dependancy.

sfuser@master:~/jzmq$ sudo make install

Making install in src

make[1]: Entering directory `/home/sfuser/jzmq/src'

make[1]: *** No rule to make target `classdist_noinst.stamp', needed by `org/zeromq/ZMQ.class'.  Stop.

make[1]: Leaving directory `/home/sfuser/jzmq/src'

make: *** [install-recursive] Error 1

Solution:

replace the classdist_noinst.stamp with classnoinst.stamp in ~/jzmq/src/Makefile.am

sed -i 's/classdist_noinst.stamp/classnoinst.stamp/g' ~/jzmq/src/Makefile.am

and ensure JAVA_HOME is set.

Apt-Get is not resolving us.archive.ubuntu.com / wget: unable to resolve host address

I had faced this issue a lot of times when i try to download / install any software, i was able to access the same URL in browser but not in terminal:

I fixed my problem by fixing a DNS address manually into file /etc/resolv.conf to DNS address of my operator. 

Previously it was set to 192.168.0.254 by Ubuntu.


$cat /etc/resov.conf

nameserver 8.8.8.8

nameserver 192.168.0.254