Openstack installation in few steps (Single Node Setup)

1. Download and Install Oracle Virtual Box


2. Download Ubuntu Operating System Image file, and create a virtual machine.

    If you face any issues then check and enable the virtualization options in your BIOS settings.


3.  While creating vm, you may end up with 2 options try Ubuntu / Install Ubuntu, click on Install Ubuntu.

4. Once the installation is over then follow the below steps to setup openstack in your vm.

                     a. sudo apt-get update

                     b. sudo apt-get upgrade

                     c. sudo apt-get install git
                     d. sudo git clone –b stable/icehouse https://github.com/openstack-dev/devstack.git

                     e. cd devstack

                     f. sudo wget –O localrc http://goo.gl/OeOGqL

                     g. ./stack.sh

now you can access openstack url http://localhost


the default credentials are:

Username  : admin / demo

password  :  openstack

NOTE : you may face some permission issues, please modify the permissions accordingly.

Setting Environment Variables with Spaces in the Path

1. We can use one temp variable to store the path with  spaces, after we can use this variable for setting the environment variables.

For ex:

to set JAVA_HOME, I can use below commands:

set temp=c:\program files\java\jre7

set JAVA_HOME="%temp%"

2. we can simply go with the below approach:


  set "JAVA_HOME=c:\program files\java\jre7"

Powershell Script to Convert xlsx to csv

Use the below code to convert the xlsx file into csv:


$xlsx_file="C:\Users\v-gibag\Desktop\locations.xlsx"
$csv_file="C:\Users\v-gibag\Desktop\user1.csv"
$Excel = new-object -comobject excel.application
$book = $Excel.Workbooks.Open($xlsx_file)
$book.SaveAs($csv_file,6)
$book.Close($false)
$Excel.quit()

How to submit hive query to HDinsight cluster from Powershell

1. Open azure power shell , please type the following command

    Get-AzurePublishSettingsFile

   It will open Azure login page, enter the credentials

 

       

Once after successful login, the Public settings file will be downloaded automatically.

2. After use the below Import command to import the settings file   

                                     

Import-AzurePublishSettingsFile <path to file>      

3.please use the below script to connect to hive and run hive.

$subscriptionName = "Azure in Open"

PS C:\> $clusterName = "IngramCluster"

PS C:\> Select-AzureSubscription $subscriptionName

PS C:\> Select-AzureSubscription -SubscriptionName $subscriptionName

PS C:\> Use-AzureHDInsightCluster $clusterName

Successfully connected to cluster IngramCluster

PS C:\> Invoke-Hive -Query "select * from hivesampletable limit 5;"

Submitting Hive query..

Started Hive query with jobDetails Id : job_1421995004773_0004

Hive query completed Successfully

8       18:54:20        en-US   Android Samsung SCH-i500        California      United States   13.9204007      0

0

23      19:19:44        en-US   Android HTC     Incredible      Pennsylvania    United States   NULL    0       0

23      19:19:46        en-US   Android HTC     Incredible      Pennsylvania    United States   1.4757422       0

1

23      19:19:47        en-US   Android HTC     Incredible      Pennsylvania    United States   0.245968        0

2

28      01:37:50        en-US   Android Motorola        Droid X Colorado        United States   20.3095339      1

                        

Calling SQL Stored procedure from R

The Following lines in R calls a stored  procedure in SQL SEREVR 

library(RODBC)
conn <- odbcDriverConnect('driver={SQL Server};server=HostName;database=DatabaseName;uid=useName;pwd=Password')
query <- paste("exec  dbo.R_getData ");
res<-sqlQuery(conn, query);

Enable oozie workflow for new mapReduce API

If you are using new MapReduce API, and wants to execute implement DAG with oozie workflow, then you
may face the below exception :

java.lang.RuntimeException: Error in configuring object
 at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
 at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
 at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.jav

Caused by: java.lang.reflect.InvocationTargetException

 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
 ... 9 more

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class TestMapper not org.apache.hadoop.mapred.Mapper
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:899)
 
Caused by: java.lang.RuntimeException: class TestMapper not org.apache.hadoop.mapred.Mapper
 at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:893)
 ... 16 more

To solve the above issue, we need to have two properties in workflow.xml,

<property>

    <name>mapred.reducer.new-api</name>

    <value>true</value>

  </property>

  <property>

    <name>mapred.mapper.new-api</name>

    <value>true</value>

</property>

now replace the workflow.xml in HDFS with updated one, the issue will be resolved.....

Accesing HBase Data using Hive Query Language ( with Probable Exceptions)

1. create a table in HBase

hbase(main):001:0> create 'hbaseTable','cf1'
0 row(s) in 1.4830 seconds

2. insert data into table

hbase(main):002:0> put 'hbaseTable','row1','cf1:name','giri'
0 row(s) in 0.0800 seconds

hbase(main):003:0> put 'hbaseTable','row2','cf1:name','Anamika'
0 row(s) in 0.0070 seconds

3. scan the table data

hbase(main):004:0> scan 'hbaseTable'
ROW                   COLUMN+CELL                                               
 row1                 column=cf1:name, timestamp=1400133482419, value=giri      
 row2                 column=cf1:name, timestamp=1400133502249, value=Anamika   
2 row(s) in 0.0360 seconds

4. Now we need to add the below jar files to hive.

guava-11.0.2.jar,
hive-hbase-handler-0.10.0.24.jar,  
hbase-0.94.5.jar, 
zookeeper-3.4.5.23.jar

we have number of ways to do this,

one way is to add jar files to

export HIVE_AUX_JARS_PATH=/usr/lib/guava-11.0.2.jar:/usr/lib/hive-hbase-handler-0.10.0.24.jar/ ...... remaining jars

other way directly add jars in the hive console,

hive> add jar /usr/lib/hbase/lib/guava-11.0.2.jar;
Added /usr/lib/hbase/lib/guava-11.0.2.jar to class path
Added resource: /usr/lib/hbase/lib/guava-11.0.2.jar

..... add remaining jars also.

now create hive table using the below syntax:

hive> CREATE TABLE hiveTable(key int, name string) 

STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,cf1:name")
TBLPROPERTIES ("hbase.table.name" = "hbaseTable");

now you can use hiveql language to query HBase data.

Troubleshooting:

You may get the below exception :

java.lang.ClassNotFoundException: org.apache.hadoop.hbase.MasterNotRunningException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)… 21 more 

or you may get any zookeeper related issues.

you can resolve the above issues by setting the below 2 properties in hive prompt;

set hbase.zookeeper.quorum=your zookeeper nodes;

set zookeeper.znode.parent=hbase-unsecure;