New release with new motion detection changes (#36)

* update readme and setup.py

* removing unused settings

* misc changes

* finish renaming scapy to kamene

* more updates to fix issues

* misc updates

* bumping version

* readme updates

* image updates

README.md

@@ -3,16 +3,12 @@
 A simple security system to run on a [Raspberry Pi](https://www.raspberrypi.org/).
 
 Features:
+
   - Motion detection and photo capture using the camera.
   - Mobile notifications with photos.
   - Detects when you are home and arms or disarms automatically.
   - Can be remotely disabled or queried using [Telegram](https://telegram.org/).
 
-Similar to these products:
-
-  - https://www.kickstarter.com/projects/vivienmuller/ulo/
-  - http://canary.is/
-
 ![rpi-security 1](../master/images/rpi-security-1.jpg?raw=true)
 
 ![rpi-security 2](../master/images/rpi-security-2.jpg?raw=true)
@@ -21,43 +17,51 @@ Similar to these products:
 ## Requirements
 
 You will need this hardware:
-  - Raspberry Pi with camera interface. I use a model A+.
+
+  - Raspberry Pi with camera interface.
   - Raspberry Pi camera module.
-  - USB Wi-Fi that supports monitor mode. I used a RT5370 based adapter, they are cheap at about €6 and easy to find.
-  - An enclosure of some sort. Details of the hardware I made is [here](hardware).
+  - A Wi-Fi adapter that supports monitor mode (see [note](#WiFi-adapter-arrangement))
+
+Software requirements:
 
-Other requirements:
-  - A [Telegram bot](https://telegram.org/blog/bot-revolution). It's free and easy to setup.
-  - Raspbian distribution installed. I used 9 (stretch). You could possibly use a different OS but I haven't tried it.
+  - A [Telegram bot](https://core.telegram.org/bots). It's free and easy to setup.
+  - Raspbian Stretch Lite distribution installed.
   - Python 3.
 
 ## How it works
 
-### Automatic presence detection
+#### Automatic presence detection
 
 One of my main goals was to have the system completely automatic. I didn't want to have to arm or disarm it when leaving or arriving home. I figured the easiest way to achieve this was to try and detect the mobile phones of the home occupants. Conceptually this was quite simple but in practice it was the most challenging part because:
+
   - Capturing all packets on a Wi-Fi interface is too resource intensive.
   - There are presently no good 5Ghz USB Wi-Fi adapters that support monitor mode. This means packet monitoring is restricted to 2.4Ghz where most modern mobile phones use 5Ghz now.
   - Mobile phones are not always online and sending packets over Wi-Fi. Sometimes they stay unconnected for 15 minutes or longer.
   - Even with 99% accuracy false alarms are annoying.
 
-After much testing I used an approach that mixes active (ARP scan) and passive (packet capture) detection over the Wi-Fi adapter based on knowing the MAC addresses of the mobile phones. The mobile phone MAC addresses are set in the configuration and the rpi-security application captures packets on a monitor mode interface with the following filter:
+After much testing I used an approach that mixes active (ARP scan) and passive (packet capture) detection over the Wi-Fi adapter based on knowing the MAC addresses of the mobile phones. The mobile phone MAC addresses are set in the configuration and rpi-security captures packets on a monitor mode interface with the following filter:
+
 1. Wi-Fi probe requests from any of the configured MACs.
 2. Any packets sent from the configured MACs to the host running rpi-security.
 
 The application resets a counter when packets are detected and if the counter goes longer than ~10 minutes the system is armed. To eliminate the many false alarms, when transitioning from armed to disarmed state or vice versa, the application performs an ARP scan directed at each of the configured MAC addresses to be sure they are definitely online or offline. Both iOS and Android will respond to this ARP scan 99% of the time where a ICMP ping is quite unreliable. By combining the capture of Wi-Fi probe requests and using ARP scanning, the Wi-Fi frequency doesn't matter because mobile phones send probe requests on both frequencies and ARP scan works across both frequencies too.
 
-### Notifications
+#### Motion detection
 
-A [Telegram](https://telegram.org/blog/bot-revolution) bot is used to send notifications with the captured images. They have good mobile applications and a nice API. You can also view the messages in a browser and messages are synced across devices.
+Motion detection is done using [OpenCV](https://opencv.org/). Each motion detection will save 4 pictures in `/tmp`:
+
+ - frame.jpg: The picture with rectangles surrounding the motion
+ - gray.jpg: The picture with grayscale and blur (which will be use to detect motion between the current frame and the previous one)
+ - abs_diff.jpg: The absolute difference between they grays frames (current and previous)
+ - thresh.jpg: A threshold has been applied to be sure the motion is important enough to be detected.
 
-If the system is in an armed state and motion is detected then a message with the captured image is sent to you from the Telegram bot.
+#### Notifications
 
-Notifications are also sent on any alarm state change.
+A [Telegram](https://core.telegram.org/bots) bot is used to send notifications with the captured images. They have good mobile applications and a nice API. You can also view the messages in a browser and messages are synced across devices. If the system is in an armed state and motion is detected then a message with the captured image is sent to you from the Telegram bot. Notifications are also sent on any alarm state change.
 
 ![rpi-security 2](../master/images/rpi-security-notification.png?raw=true)
 
-### Remote control
+#### Remote control
 
 You can send the Telegram bot commands that trigger certain actions.
 
@@ -67,69 +71,73 @@ You can send the Telegram bot commands that trigger certain actions.
   - */photo*: Captures and sends a photo.
   - */gif*: Captures and sends a gif.
 
-![rpi-security 4](../master/images/rpi-security-status-message.png?raw=true)
+#### Python
 
-### Python
+The application is written in python 3 and large parts of the functionality are provided by the following pip packages:
 
-The application is written in python 3. Large parts of the functionality are provided by the following pip modules:
   - [picamera](https://github.com/waveform80/picamera)
-  - [scapy](http://www.secdev.org/projects/scapy/)
+  - [kamene](https://github.com/phaethon/kamene)
   - [python-telegram-bot](https://github.com/python-telegram-bot/python-telegram-bot)
+  - [opencv-python](https://github.com/skvark/opencv-python)
 
 The application uses multithreading in order to process events asynchronously. There are 4 threads:
+
   - telegram_bot: Responds to commands.
   - monitor_alarm_state: Arms and disarms the system.
   - capture_packets: Captures packets from the mobile devices.
   - process_photos: Sends captured images via Telegram messages.
 
-### Motion detection
-Previously, the motion detection was made by the default RpiMotionDetector from RpiCamera which could cause many false positive.
-In the newer version, the motion detection is done thanks to OpenCV and Image algorithm, reducing the number of false positive.
-Each motion detection will save 4 pictures in /tmp:
- - frame.jpg: The picture with rectangles surrounding the motion
- - gray.jpg: The picture with grayscale and blur (which will be use to detect motion between the current frame and the previous one)
- - abs_diff.jpg: The absolute difference between they grays frames (current and previous)
- - thresh.jpg: A threshold has been applied to be sure the motion is important enough to be detected.
-
 ## Installation, configuration and Running
 
-The interface used to connect to your WiFi network must be the same interface that supports monitor mode. And this must be the same WiFi network that the mobile phones connect to.
+First ensure your WiFi is [set up correctly](#WiFi-adapter-arrangement))
 
-First install required packages:
+Install required packages:
 
-```
-sudo apt-get update
-sudo apt-get install -y tcpdump iw python3-dev python3-pip libjpeg8-dev zlib1g-dev libffi-dev python3-numpy libopenjp2-7-dev libtiff5 install libatlas-base-dev libjasper-dev libqtgui4 python3-pyqt5 libqt4-test
-sudo pip3 install --upgrade pip
+```console
+sudo apt update
+sudo apt install -y libhdf5-100 libharfbuzz0b libwebp6 libjasper1 libilmbase12 libopenexr22 libgstreamer1.0-0 libavcodec-extra57 libavformat57 libswscale4 libgtk-3-0 libqtgui4 libqt4-test libatlas-base-dev tcpdump iw python3-dev python3-pip libjpeg8-dev zlib1g-dev libffi-dev python3-numpy libopenjp2-7-dev libtiff5
 ```
 
-Install rpi-security, reload systemd configuration and enable the service:
+Install open-cv and rpi-security:
 
+```console
+sudo pip3 install opencv-contrib-python opencv-contrib-python-headless
+sudo pip3 install --no-binary :all: https://github.com/FutureSharks/rpi-security/archive/1.0.zip
 ```
-sudo pip3 install --no-binary :all: https://github.com/FutureSharks/rpi-security/archive/master.zip
+
+Reload systemd configuration and enable the service:
+
+```console
 sudo systemctl daemon-reload
 sudo systemctl enable rpi-security.service
 ```
 
-Add your MAC address or addresses, Telegram bot API key and any other changes to ``/etc/rpi-security.conf``.
+Add your MAC address or addresses, Telegram bot API key and any other changes to `/etc/rpi-security.conf`.
 
-Ensure you have enabled the camera module using ``raspi-config``.
+Ensure you have enabled the camera module using `raspi-config`.
 
 And start the service:
 
-```
+```console
 sudo systemctl start rpi-security.service
 ```
 
-You need to send at least one message to the Telegram bot otherwise it won't be able to send you messages. This is so the service can save the telegram chat_id. So just send the ``/status`` command.
+You need to send at least one message to the Telegram bot otherwise it won't be able to send you messages. This is so the service can save the telegram chat_id. So just send the `/status` command.
 
-It runs as a service and logs to syslog. To see the logs check ``/var/log/syslog``.
+It runs as a service and logs to syslog. To see the logs check `/var/log/syslog`.
 
-There is also a debug option that logs to stdout:
+## Debug and troubleshooting
 
-```
+You can start `rpi-security.py` manually with debug output. First add the monitor mode interface:
+
+```console
 root@raspberrypi:~# iw phy phy0 interface add mon0 type monitor
 root@raspberrypi:~# ifconfig mon0 up
+```
+
+Then start with debug output:
+
+```console
 root@raspberrypi:~# rpi-security.py -d
 2016-05-28 14:43:30 DEBUG   rpi-security.py:73  MainThread          State file read: /var/lib/rpi-security/state.yaml
 2016-05-28 14:43:30 DEBUG   rpi-security.py:44  MainThread          Calculated network: 192.168.178.0/24
@@ -144,11 +152,29 @@ root@raspberrypi:~# rpi-security.py -d
 2016-05-28 14:44:48 DEBUG   rpi-security.py:280 Dummy-1             Motion detected but current_state is: disarmed
 ```
 
-This is all that is required on my Raspberry Pi Model A+.
+And then delete the interface when complete:
 
-This shows my WLAN network device arrangement:
+```console
+iw dev mon0 del
+```
+
+## WiFi adapter arrangement
+
+Your WLAN or WiFi adapter must support monitor mode. The Raspberry Pi built-in wireless LAN adapters do **not** support monitor mode by default. Currently the only way to get monitor mode working for the built-in WiFi adapters is to use [nexmon](https://github.com/seemoo-lab/nexmon) and this is not simple.
+
+The easiest way to get a monitor mode WiFi adapter is to just buy a RT5370 based adapter. They are cheap at about €6 and easy to find.
+
+The interface used to connect to your WiFi network must be the same interface that supports monitor mode. And this must be the same WiFi network that the mobile phones connect to. This is because there is a packet capture running to listen for mobile phone ARP replies and Wi-Fi probe requests.
 
+If you are using a USB adapter and want to disable your onboard WiFi adapter run this and reboot:
+
+```console
+sudo sh -c "echo 'blacklist brcmfmac\nblacklist brcmutil' > /etc/modprobe.d/disable-onboard-wifi.conf"
 ```
+
+This shows a working WiFi adapter arrangement:
+
+```console
 root@raspberrypi:~# iw dev
 phy#0
         Interface mon0
@@ -164,18 +190,18 @@ phy#0
                 channel 1 (2412 MHz), width: 40 MHz, center1: 2422 MHz
 ```
 
-You could have interfaces with different names, just be sure to change the ``network_interface`` parameter in ``/etc/rpi-security.conf`` and also the reference to mon0 in [rpi-security.service](https://github.com/FutureSharks/rpi-security/blob/master/etc/rpi-security.service)
+You could have interfaces with different names, just be sure to change the `network_interface` parameter in `/etc/rpi-security.conf` and also the reference to `mon0` in [rpi-security.service](https://github.com/FutureSharks/rpi-security/blob/master/etc/rpi-security.service)
 
-### Older version with PIR sensor motion detection
+## Older version with PIR sensor motion detection
 
-Currently the camera is used for motion detection. If you want to use the old version with support for a PIR sensor then look at version 0.7:
+Currently the camera is used for motion detection. If you want to use the old version with support for a PIR sensor then look at version `0.7`:
 
-https://github.com/FutureSharks/rpi-security/releases/tag/0.7
+https://github.com/FutureSharks/rpi-security/tree/0.7
 
-### Reboot on connectivity loss
+## Reboot on connectivity loss
 
 About once every month or two my Raspberry Pi loses the WLAN connection. I created a cron job to check connectivity and reboot if the check fails.
 
-```
+```console
 echo '*/20 * * * * root /usr/bin/host api.telegram.org > /dev/null 2>1 || (/usr/bin/logger "Rebooting due to connectivity issue"; /sbin/shutdown -r now)' > /etc/cron.d/reboot-on-connection-failure
 ```

TODO.md

@@ -5,4 +5,5 @@
 - Update photos now that PIR sensor is not required
 - Tidy up comments
 - Remove PIR CAD files
+- Fig syslog formatting `Dec 22 10:40:02 raspberrypi monitor_alarm_state.py:...` should include service name
 - Don't save motion detection related photos to the filesystem, [save directly to OpenCV object](https://picamera.readthedocs.io/en/release-0.6/recipes.html#capturing-to-an-opencv-object)

bin/rpi-security.py

@@ -48,8 +48,8 @@ def setup_logging(debug_mode, log_to_stdout):
     try:
         rpis = rpisec.RpisSecurity(args.config_file, args.data_file)
         camera = rpisec.RpisCamera(rpis.photo_size, rpis.gif_size, rpis.motion_size,
-            rpis.camera_vflip, rpis.camera_hflip, rpis.motion_detection_setting,
-            rpis.camera_capture_length, rpis.camera_mode)
+            rpis.camera_vflip, rpis.camera_hflip, rpis.camera_capture_length,
+            rpis.camera_mode)
         if rpis.debug_mode:
             logger.handlers[0].setLevel(logging.DEBUG)
     except Exception as e:

etc/rpi-security.conf

@@ -5,12 +5,12 @@ mac_addresses=aa:aa:aa:bb:bb:bb,cc:cc:cc:dd:dd:dd
 # The Telegram bot token.
 telegram_bot_token=999999999:xxxxxxx-zzzzzzzzzzzzzzzzz
 
-# The wireless interface in monitor mode
-network_interface=mon0
-
 # The number of people allowed to chat with Telegram bot
 telegram_users_number=1
 
+# The wireless interface in monitor mode
+network_interface=mon0
+
 # Debug mode. Setting this to true will send more verbose logging to syslog
 debug_mode=false
 
@@ -31,7 +31,7 @@ camera_hflip=false
 
 # Number of photos to take or number of GIF frames x3 when motion is detected.
 # Higher than 3 with gif mode will run out of memory on models with 128MB of RAM.
-camera_capture_length=3
+camera_capture_length=2
 
 # Image size for photos
 photo_size=2592x1944
@@ -41,6 +41,3 @@ gif_size=800x600
 
 # Resolution for motion detection
 motion_size=1280x720
-
-# Motion detection settings: motion_magnitude x motion_vectors. Higher of either setting means less sensitivity. Requires some experimentation.
-motion_detection_setting=60x17

hardware/README.md

@@ -1,15 +1,13 @@
 # Raspberry Pi Security System Hardware
 
-I used a [Raspberry Pi Model A+](https://www.raspberrypi.org/products/model-a-plus/) because it's the smallest pi with the camera interface. The frame is made with [Makerbeam](http://www.makerbeam.com/), 4x vertical 66mm beams and 8x horizontal 54mm beams.
+I used a [Raspberry Pi 3 Model A+](https://www.raspberrypi.org/products/raspberry-pi-3-model-a-plus/). The frame is made with [Makerbeam](http://www.makerbeam.com/), 4x vertical 66mm beams and 8x horizontal 54mm beams.
 
 Raspberry Pi camera mount 3D printed from [Shapeways](https://www.shapeways.com/).
 
 DWG CAD files are in hardware/cad-files or you can see models here on Spaceways:
 
 https://www.shapeways.com/product/K6FLBY3SW/raspberry-pi-camera-module-makerbeam-mount
 
-https://www.shapeways.com/product/F4268ZMG5/raspberry-pi-pir-sensor-makerbeam-mount
-
 The camera mount supports both the V1 and V2 camera modules.
 
 ## Photos
@@ -18,4 +16,8 @@ The camera mount supports both the V1 and V2 camera modules.
 
 ![rpi-security 2](../../master/images/rpi-security-2.jpg?raw=true)
 
-![rpi-security 3](../../master/images/rpi-security-hardware-1.jpg?raw=true)
+![rpi-security 3](../../master/images/rpi-security-3.jpg?raw=true)
+
+![rpi-security 4](../../master/images/rpi-security-4.jpg?raw=true)
+
+![rpi-security 5](../../master/images/makerbeam-camera-bracket.png?raw=true)

rpisec/rpis_camera.py

@@ -26,16 +26,14 @@ class RpisCamera(object):
     captues photos and GIFs.
     '''
     def __init__(self, photo_size, gif_size, motion_size, camera_vflip,
-            camera_hflip, motion_detection_setting, camera_capture_length,
+            camera_hflip, camera_capture_length,
             camera_mode):
         self.photo_size = photo_size
         self.gif_size = gif_size
         self.camera_vflip = camera_vflip
         self.camera_hflip = camera_hflip
         self.lock = Lock()
         self.queue = Queue()
-        self.motion_magnitude = motion_detection_setting[0]
-        self.motion_vectors = motion_detection_setting[1]
         self.motion_framerate = 5
         self.motion_size = motion_size
         self.temp_directory = '/var/tmp'
@@ -47,7 +45,6 @@ def __init__(self, photo_size, gif_size, motion_size, camera_vflip,
             self.camera = PiCamera()
             self.camera.vflip = self.camera_vflip
             self.camera.hflip = self.camera_hflip
-            self.camera.led = False
         except Exception as e:
             exit_error('Camera module failed to intialise with error {0}'.format(repr(e)))
 

rpisec/rpis_security.py

@@ -6,12 +6,12 @@
 import yaml
 import logging
 
-logging.getLogger("scapy.runtime").setLevel(logging.ERROR)
+logging.getLogger("kamene.runtime").setLevel(logging.ERROR)
 
 from configparser import SafeConfigParser
 from netaddr import IPNetwork
 from netifaces import ifaddresses
-from scapy.all import srp, Ether, ARP
+from kamene.all import srp, Ether, ARP
 from telegram import Bot as TelegramBot
 from .exit_clean import exit_error
 from .rpis_state import RpisState
@@ -35,9 +35,9 @@ class RpisSecurity(object):
         'photo_size': '1024x768',
         'gif_size': '1024x768',
         'motion_size': '1024x768',
-        'motion_detection_setting': '60x10',
         'camera_mode': 'gif',
-        'camera_capture_length': '3'
+        'camera_capture_length': '3',
+        'telegram_users_number': '1'
     }
 
     def __init__(self, config_file, data_file):
@@ -131,7 +131,6 @@ def _str2bool(v):
         self.photo_size = tuple([int(x) for x in self.photo_size.split('x')])
         self.gif_size = tuple([int(x) for x in self.gif_size.split('x')])
         self.motion_size = tuple([int(x) for x in self.motion_size.split('x')])
-        self.motion_detection_setting = tuple([int(x) for x in self.motion_detection_setting.split('x')])
         self.camera_capture_length = int(self.camera_capture_length)
         self.camera_mode = self.camera_mode.lower()
         self.packet_timeout = int(self.packet_timeout)