[RPi] Hadoop 在 Raspberry Pi 上使用

在 Pi 上使用 Hadoop 常會遇到這樣的 warning

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable 

是因為預設的 Hadoop 執行檔使用 32-bit x86 native libraries, 如果要解決這 warning 就要自己 build, 步驟如下

安裝 maven 還有很多後面編譯會用到的 lib

sudo apt-get install -y maven build-essential g++ autoconf automake libtool cmake zlib1g-dev pkg-config libssl-dev libfuse-dev libsnappy-dev libsnappy-java libbz2-dev

使用的 Java 版本

java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) Client VM (build 25.65-b01, mixed mode)

安裝 protobuf 2.5.0

wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz
tar zxvf protobuf-2.5.0.tar.gz
cd protobuf-2.5.0 
./configure --prefix=/usr
make 
sudo make install 

下載 Hadoop 與 patch

cd ~
wget ftp://apache.belnet.be/mirrors/ftp.apache.org/hadoop/common/hadoop-2.7.1/hadoop-2.7.1-src.tar.gz
tar -xvzf hadoop-2.7.1-src.tar.gz 

安裝 patch

cd hadoop-2.7.1-src/hadoop-common-project/hadoop-common/src 
wget https://issues.apache.org/jira/secure/attachment/12570212/HADOOP-9320.patch 
patch < HADOOP-9320.patch  

編譯 Hadoop, 在 Pi 3 大概要等一個半小時左右, 預設會安裝到 /opt 目錄下

cd ../../.. 
mvn package -Pdist,native -DskipTests -Dtar 

將 native lib 複製到

cd hadoop-dist/target/hadoop-2.7.1/lib/native/ 
cp * /opt/hadoop-2.7.1/lib/native/  

 之後就和一般的 Hadoop 使用設定一樣了

reference:
* Building the native Hadoop library for Raspberry Pi's ARMv7 processor

[RPi] Hadoop

一些連結, 如果下次還會用到再來看吧

* What You Need To Know About Hadoop and Its Ecosystem
* Platform base: HDFS + MR
* 5. MapReduce, Hadoop
* Hadoop 新 MapReduce 框架 Yarn 詳解
* 海量數據中的SMAQ技術
* HOW TO SETUP A CHEAP MULTI-NODE HADOOP CLUSTER
* MapReduce Tutorial

[RPi] Docker

感謝 Philip 大大加持, 剛剛又看到一篇好文, 好像是我按的第一個讚吧？

How to deploy PHP projects with docker

[RPi] Raspberry Pi藍牙程式(Python)

使用RFCOMM Socket連線

#!/usr/bin/python

import bluetooth
import thread
import time

server_sock=bluetooth.BluetoothSocket( bluetooth.RFCOMM )
print "bluetooth.RFCOMM: " + str(bluetooth.RFCOMM)

port = 1
server_sock.bind(("",port))
server_sock.listen(1)

client_sock,address = server_sock.accept()
print "Accepted connection from ", address

def receive_thread():
    while True:
        try:
            data = client_sock.recv(1024)
            print "received [%s]" % data

        except:
            print "SerialException: "
        time.sleep(0.1)

thread.start_new_thread(receive_thread, ())

while True:
    s = raw_input("RPi] ")
    client_sock.send(s)

client_sock.close()
server_sock.close()

使用RFCOMM Serial連線

#!/usr/bin/python

import serial
import time
import thread

port = serial.Serial("/dev/rfcomm0", baudrate=38400, timeout=0.1)

def receive_thread():
    while True:
        try:
            recvS = port.readlines()
            for recv in recvS:
                print("\nAndroid] " + recv.replace("\n", ""))

        except serial.SerialException:
            print "SerialException: "
            port.close()

        time.sleep(0.1)

thread.start_new_thread(receive_thread, ())

while True:
    s = raw_input("RPi] ")
    port.write(s)

[RPi] Pi 3 + Arduino Bluetooth with HC-05

之前測了 Pi 3 和 Android 間的藍牙通訊

現在再測了 Pi 3 和 HC-05 的通訊, 算是打通藍牙觀念了

這兩篇是最後的一哩路
https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=163712
https://www.raspberrypi.org/forums/viewtopic.php?p=947185

[Ubuntu] Debian based package management

這張圖找好久, 果然還是有好心人畫, 感謝

reference: https://hughchao.hackpad.com/ep/pad/static/wrP2bgGpbe5

[阿宅專區] 藍牙歷史

一些藍牙歷史名詞可參考

藍牙
Bluetooth
* Bluetooth Versions Walkthrough, and Bluetooth 4.0 Low Energy Development Resources
藍芽藍牙傻傻分不清楚
藍牙新標誌，smart & smart ready
* 藍牙協議分析(1)_基本概念
* 藍牙協議分析(2)_協議架構
* 藍牙協議分析(3)_藍牙低功耗(BLE)協議棧介紹
http://www.wowotech.net/sort/bluetooth
值得一讀與分享的BT 4.0 (BLE)低功耗藍芽介紹
* Bluetooth - How it works
* Bluetooth Basics
* Connectivity of the Internet of Things

3
蓝牙3.0
藍牙多模應用與認證規範面面觀

4.1
解析如何選擇藍牙4.1和藍牙4.0？
瞄準物聯網應用　藍牙4.1版功能全面升級
藍牙4.1深入詳解：其實並不遙遠

4.2
Bluetooth LE 4.0 / 4.1 / 4.2 新增功能簡介

BLE
* Introduction to Bluetooth Low Energy (Adafruit)
Bluetooth Low Energy (BLE) 是什麼?

科普
[置頂] 享受藍牙(一): 開始享受

分析封包
假日藍芽工程師, 先動起來再說
hcitool and gatttool example

[RPi] Raspberry Pi 3 Bluetooth Beacon

兩行就可以了

sudo hciconfig hci0 leadv 3
sudo hcitool -i hci0 cmd 0x08 0x0008 18 02 01 06 03 03 aa fe 10 16 aa fe 10 00 00 65 64 75 70 6f 6c 69 2e 66 69 00 00 00 00 00 00 00

欄位的意義可參考這篇
http://blog.itist.tw/2014/05/ibeacon.html

這篇的欄位更清楚
https://hackaday.io/project/10314-raspberry-pi-3-as-an-eddystone-url-beacon


使用 physical web, iOS 讀取 notification 沒問題, 但 Android 反而讀不到, 可以參考這篇的作法(雖然目前還是沒收到...)
https://community.estimote.com/hc/en-us/articles/218451777-How-to-enable-Physical-Web-in-Chrome-

這篇是 Pi 3 + Android(Node.js) 的應用
https://www.hackster.io/inmyorbit/build-a-mobile-app-that-connects-to-your-rpi-3-using-ble-7a7c2c

實做 physical web(eddystone-url) 的套件
Node.js: https://github.com/don/node-eddystone-beacon
Python: https://github.com/google/eddystone/tree/master/eddystone-url/implementations/PyBeacon

[Python] decorator

在 python 很常用到

這幾篇寫得蠻清楚的,
http://thecodeship.com/patterns/guide-to-python-function-decorators/
http://missions5.blogspot.tw/2014/07/python-decorator.html

[RPi] 簡單的教學

還不錯, 寫成這樣就可以了
http://workshop.raspberrypiaustralia.com/

[Android] AndroidScreencast

因為 ubuntu 版本是 10.04 LTS 太舊了, 所以一堆問題要解決

要把 Android 的畫面打到 ubuntu 應該要有一堆方法, 記得之前用 Droid@Screen, 可是現在不知道怎麼了不能用

後來又看到 AndroidScreencast, 雖然感覺有點慢也將就著用吧

$ git clone https://github.com/xSAVIKx/AndroidScreencast
$ cd AndroidScreencast
$ mvn package

馬上報錯, 因為沒有裝 maven

$ sudo apt-get install maven2

結果軟體最低限制 Java 7 and Maven 3.2.5, 只好想辦法裝 maven3

因為沒有 apt, 只好抓 binary, 參考這篇文章, 可是現在只剩下 maven-3.2.2, 就加減用吧

再來是修改環境變數 JAVA_HOME, M2_HOME, MAVEN_HOME, M2 一堆

修改完還要把 maven2 移除掉才行

$ sudo apt-get autoremove maven2

確認版本
$ mvn -v
Apache Maven 3.2.2

終於可以編譯了

$ mvn package

編譯完成後執行發現又錯, 錯誤訊息在 app.properties, 要把

adb.path=adb.exe

改為

adb.path=adb

記得 adb 也要 export 到環境變數, 我是用鳥鳥的 appinventor 帶的 adb

最後終於可以執行了

$ java -jar target/androidscreencast-0.0.7s-executable.jar

唉, 舊版本就是該死

[RPi] 藍牙通訊

硬體：Pi 3
軟體：2016-05

Android 手機安裝 BlueTerm, 原始碼

修改 /etc/systemd/system/dbus-org.bluez.service, 將

ExecStart=/usr/lib/bluetooth/bluetoothd

改為

ExecStart=/usr/lib/bluetooth/bluetoothd --compat

重新載入設定檔
$ sudo systemctl daemon-reload


Pi 3 藍牙設定
$ sudo bluetoothctl
[bluetoothctl] power on
[bluetoothctl] scan on
[bluetoothctl] agent on
[bluetoothctl] pair MAC_ADDRESS
[bluetoothctl] trust MAC_ADDRESS
[bluetoothctl] exit
sudo rfcomm bind 0 MAC_ADDRESS


Pi 3 程式碼：

#!/usr/bin/python

import bluetooth
import thread
import time

server_sock=bluetooth.BluetoothSocket( bluetooth.RFCOMM )

port = 1 
server_sock.bind(("",port))
server_sock.listen(1)

client_sock,address = server_sock.accept()
print "Accepted connection from ", address

def receive_thread():
    while True:
        try:
            data = client_sock.recv(1024)
            print "received [%s]" % data

        except:
            print "SerialException: "

        time.sleep(0.1)

thread.start_new_thread(receive_thread, ()) 

while True:
    s = raw_input("RPi] ")
    client_sock.send(s)

client_sock.close()
server_sock.close()

reference:
* An Introduction to Bluetooth Programming
* Using smart phone (Android) to control LED on Raspberry Pi 3 through Bluetooth connection


如果透過 App Inventor 寫 Android 端程式, 可以參考這篇文章
http://howtomechatronics.com/tutorials/arduino/how-to-build-custom-android-app-for-your-arduino-project-using-mit-app-inventor/

目前還沒搞定, 之後會再打通這邊...
列出有用到的 PORT

import serial.tools.list_ports

ports = list(serial.tools.list_ports.comports())
for p in ports:
    print p