Become a FAN of VRSEC SOURCES BECOME A FAN<<>> updates:autobiography; novels<<>>subscibe to Asist_sources mobile alerts & get daily updates of this blog to ur mobile along with motivational quotes ,funny sms , frienship sms and many more.

Search The Blog

Showing posts with label spark. Show all posts
Showing posts with label spark. Show all posts

Best Commands to find the information about the LInux OS running

5

uname 
uname -a
uname -v
uname --help
cat /etc/issue.net
cat /etc/redhat-release
lsb_release -a   ==> I prefer this command 

Unable to open kernel device "\\.\Global\vmx86": The system cannot find the file

0
Unable to open kernel device
"
.\Global\vmx86": The system cannot find the file specified. Did you reboot
after installing VMware Workstation?
Failed to initialize



Solution for this type error which occurred when opening an vmplayer. 



try to re-install the vmx86 driver

open a cmd in adminmode and navigate to the VMware installation directory and run

 vnetlib -- uninstall vmx86 
reboot
check again with the net start command - this time it should say "service name  is invalid"
then run
 vnetlib -- install vmx86 
and reboot again

now it hopefully works


Second simple solutions is 

start CMD as administrator then run  "net start vmx86".




Start a spark master worker nodes spark-shell in Windows 10

0

open cmd 
goto the installed folder of spark


C:\Spark>bin\spark-class.cmd org.apache.spark.deploy.master.Master

C:\Spark\bin>spark-class org.apache.spark.deploy.worker.Worker spark://192.168.5
6.1:7077



Now Another command prompt type : 

spark-shell --master spark://192.168.56.1:7077


Now the screen shot is


Now you can see the Application ID . Click on the Id and explore yourself.

Be sure top have SPARK_HOME is set in environmental variables ..


Note:
we have to open different cmd's to execute commands
cntrl+c on command line to terminate the process

Spark Windows Installation ERRORS : WINUTILS.EXE , RuntimeException

0
Spark Installation in Windows caused two errors to start successfully without any errors. 

1. Missing binary file WINUTILS.EXE

  Solution :  Download the latest WINUTILS.EXE and place that in %HADOOP_HOME%\bin\winutils.exe 

Note: winutils.exe  should be more carefull in downloading the LATEST one


 prechecks before downloading :
  1. check whether windows is 32 or 64 bit 
  2.checking the winutils downloadable is 32 or 64 bit compatable 

Because 32 bit will work for 32 bit and 64 bit will work for 64 bit only 


2. ava.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /t
mp/hive on HDFS should be writable. Current permissions are: rw-rw-rw-


After donwloading and importing the winutils.exe in bin folder .

create /tmp/hive  folder in C:\  using command line in admin mode 

and run the below commands 

%HADOOP_HOME%\BIN\WINUTILS.EXE chmod -R 777 \tmp\hive  
%HADOOP_HOME%\BIN\WINUTILS.EXE ls \tmp\hive 

At last you must contain C:\tmp\hive with full access to every user



in my case 

SPARK_HOME = C:\Spark\bin
HADOOP_HOME=C:\Spark


Thanks for reading

Disclaimer

This is a cool blog where we can find many of our stuff.Like me who want to contribute to this blog sign in with google account appearing on the left side(below subscribe blog).Then u can post any stuff to help our frnds.
thank u frnds.


To help u how to roam on this site ?how to check ur topics?
see the menu and u will find links which appeared in blue and click on the option u need(appeared below cheeky quotes).
or
see the blog archieve (below the search blog).