Exception in thread "main" java.lang.NoSuchFieldError: IBM_JAVA

WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread "main" java.lang.NoSuchFieldError: IBM_JAVA
        at org.apache.hadoop.security.UserGroupInformation.getOSLoginModuleName(UserGroupInformation.java:365)
        at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:410)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2812)
        at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2668)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
        at Write.main(Write.java:14)

Solution:

This is the problem of conflict b/w hadoop-core-*.jar and hadoop-auth-*.jar.

I have resolved this issue by removing hadoop-core-*.jar from my CLASSPATH.

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

hadoop@master:~$ hadoop fs -chown -R sfuser /storm

15/05/08 06:10:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

This is just a warning, it will not create any problems.

If you want to eliminate this warning just export the below native dir

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=/home/hadoop/hadoop-2.7.0/lib/native/"

hadoop@master:~$ hadoop fs -chown -R sfuser /storm
hadoop@master:~$

backtype.storm.StormSubmitter - Topology submission exception: Field storm.zookeeper.servers must be an Iterable of java.lang.String

While submitting storm topology on production cluster, if you mention zookeeper server as mentioned below then it will through the exception

        Config conf = new Config();
        conf.put(Config.NIMBUS_HOST, "nimbus-ip");
        conf.put(Config.NIMBUS_THRIFT_PORT,6627);
        conf.put(Config.STORM_ZOOKEEPER_PORT,2181);
        conf.put(Config.STORM_ZOOKEEPER_SERVERS,"zkerver-ip");
 
3126 [main] WARN  backtype.storm.StormSubmitter - Topology submission exception: Field storm.zookeeper.servers must be an Iterable of java.lang.String
Exception in thread "main" InvalidTopologyException(msg:Field storm.zookeeper.servers must be an Iterable of java.lang.String)
        at backtype.storm.generated.Nimbus$submitTopology_result.read(Nimbus.java:2466)
        at org.apache.thrift7.TServiceClient.receiveBase(TServiceClient.java:78)
        at backtype.storm.generated.Nimbus$Client.recv_submitTopology(Nimbus.java:162)
        at backtype.storm.generated.Nimbus$Client.submitTopology(Nimbus.java:146)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:127)
        at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
        at com.cisco.PrimeNumberTopology.main(PrimeNumberTopology.java:40)

Solution:

  

       create a list of zookeeper objects

        List<String> li=new ArrayList<String>();
        li.add("zkserver-1");

        and change the above configuration line:
       
        conf.put(Config.STORM_ZOOKEEPER_SERVERS,li);



      
         
         

make[1]: *** No rule to make target `classdist_noinst.stamp', needed by `org/zeromq/ZMQ.class'.

You may face this issue while installing jzmq as storm node dependancy.

sfuser@master:~/jzmq$ sudo make install

Making install in src

make[1]: Entering directory `/home/sfuser/jzmq/src'

make[1]: *** No rule to make target `classdist_noinst.stamp', needed by `org/zeromq/ZMQ.class'.  Stop.

make[1]: Leaving directory `/home/sfuser/jzmq/src'

make: *** [install-recursive] Error 1

Solution:

replace the classdist_noinst.stamp with classnoinst.stamp in ~/jzmq/src/Makefile.am

sed -i 's/classdist_noinst.stamp/classnoinst.stamp/g' ~/jzmq/src/Makefile.am

and ensure JAVA_HOME is set.

Apt-Get is not resolving us.archive.ubuntu.com / wget: unable to resolve host address

I had faced this issue a lot of times when i try to download / install any software, i was able to access the same URL in browser but not in terminal:

I fixed my problem by fixing a DNS address manually into file /etc/resolv.conf to DNS address of my operator. 

Previously it was set to 192.168.0.254 by Ubuntu.


$cat /etc/resov.conf

nameserver 8.8.8.8

nameserver 192.168.0.254

Openstack installation in few steps (Single Node Setup)

1. Download and Install Oracle Virtual Box


2. Download Ubuntu Operating System Image file, and create a virtual machine.

    If you face any issues then check and enable the virtualization options in your BIOS settings.


3.  While creating vm, you may end up with 2 options try Ubuntu / Install Ubuntu, click on Install Ubuntu.

4. Once the installation is over then follow the below steps to setup openstack in your vm.

                     a. sudo apt-get update

                     b. sudo apt-get upgrade

                     c. sudo apt-get install git
                     d. sudo git clone –b stable/icehouse https://github.com/openstack-dev/devstack.git

                     e. cd devstack

                     f. sudo wget –O localrc http://goo.gl/OeOGqL

                     g. ./stack.sh

now you can access openstack url http://localhost


the default credentials are:

Username  : admin / demo

password  :  openstack

NOTE : you may face some permission issues, please modify the permissions accordingly.

Setting Environment Variables with Spaces in the Path

1. We can use one temp variable to store the path with  spaces, after we can use this variable for setting the environment variables.

For ex:

to set JAVA_HOME, I can use below commands:

set temp=c:\program files\java\jre7

set JAVA_HOME="%temp%"

2. we can simply go with the below approach:


  set "JAVA_HOME=c:\program files\java\jre7"

Powershell Script to Convert xlsx to csv

Use the below code to convert the xlsx file into csv:


$xlsx_file="C:\Users\v-gibag\Desktop\locations.xlsx"
$csv_file="C:\Users\v-gibag\Desktop\user1.csv"
$Excel = new-object -comobject excel.application
$book = $Excel.Workbooks.Open($xlsx_file)
$book.SaveAs($csv_file,6)
$book.Close($false)
$Excel.quit()

How to submit hive query to HDinsight cluster from Powershell

1. Open azure power shell , please type the following command

    Get-AzurePublishSettingsFile

   It will open Azure login page, enter the credentials

 

       

Once after successful login, the Public settings file will be downloaded automatically.

2. After use the below Import command to import the settings file   

                                     

Import-AzurePublishSettingsFile <path to file>      

3.please use the below script to connect to hive and run hive.

$subscriptionName = "Azure in Open"

PS C:\> $clusterName = "IngramCluster"

PS C:\> Select-AzureSubscription $subscriptionName

PS C:\> Select-AzureSubscription -SubscriptionName $subscriptionName

PS C:\> Use-AzureHDInsightCluster $clusterName

Successfully connected to cluster IngramCluster

PS C:\> Invoke-Hive -Query "select * from hivesampletable limit 5;"

Submitting Hive query..

Started Hive query with jobDetails Id : job_1421995004773_0004

Hive query completed Successfully

8       18:54:20        en-US   Android Samsung SCH-i500        California      United States   13.9204007      0

0

23      19:19:44        en-US   Android HTC     Incredible      Pennsylvania    United States   NULL    0       0

23      19:19:46        en-US   Android HTC     Incredible      Pennsylvania    United States   1.4757422       0

1

23      19:19:47        en-US   Android HTC     Incredible      Pennsylvania    United States   0.245968        0

2

28      01:37:50        en-US   Android Motorola        Droid X Colorado        United States   20.3095339      1