Loxone UDP/HTTP Command Parser Syntax

Loxone can parse incoming data e.g. via UDP/HTTP to detect variables. I found their documentation not really good. It feels like it is almost an art to setup these parse strings, while it is actually quite simple.

The string is searched in the incoming data stream. If the full string is detected, it succeeds and returns the value (0.0 being the default). If it didn’t match the whole pattern within the stream, it resets and continues again to parse the data. Only if the whole string was parsed fully, will the matching stop.

Possible characters in the parse string

Matching individual characters

  • 7-bit ASCII characters: they are taken as-is and need to match
  • \\ matches a single \ (0x5C)
  • \n matches a LF (0x0A)
  • \r matches a CR (0x0D)
  • \t matches a TAB (0x09)
  • \xAB match a hex byte AB. This allows matching non-ASCII characters. \xAB would match 0xAB.
  • \a match a letter (A-Z, a-z)
  • \b match a TAB or space (0x09, 0x20)
  • \m match a letter or digit (A-Z, a-z, 0-9)
  • \d match a digit (0-9)
  • \. matches any byte (= skips/ignores one byte)

Matching multiple characters

  • \# match any number digits and . , or -. This should match a regular floating point number and continue after the number.
  • \w match any number of letters and digits (A-Z, a-z, 0-9). This should match a word and continue after that word.
  • \s123 skips/ignores 123 bytes in the incoming data stream. All digits following \s are using to build a number, the value can be really large – more than is ever needed.

Searching

  • \iXXX\i searches the string XXX and continues after that string with matching. The string can have standard UNIX control characters (\a, \b, \f, \n, \r, \t, \v plus the hex extension: \xAB).

Storing a value

The returned value is always a 64-bit floating point number, which can result in rounding issues for certain integer values.

  • \1…\8 stores the following byte as part of a 64-bit binary integer result. \1 is the LSB, \8 is the MSB. Sign-extension can be applied.
  • \h (hex value) stores the following ASCII-hex data (0-9, a-f, A-F) as a 32-bit MSB integer result. First invalid character ends the value. The value is returned as-is, no sign extension is applied.
  • \v (value) stores a value created from ASCII characters. Any number of spaces before the values are ignored. Then the optional sign (+ or -) can follow, plus more spaces. A &nbps; is ignored after the sign as well. The number is any number of digits (0-9) followed by (, or .) plus more digits. Then an exponent (E or e) can follow plus more digits. The first character not matching the described number will end the value. The resulting number is obviously a floating point value.
  • If \f (factor?) is part of the search term, the value will always be 0.0. This feels like a bug in the code and \f was probably thought to be a multiplication factor for the incoming value.

It is possible to have several \h and \v in the search pattern. Only the last one will be used. Same with \1, etc. If more than one \1 occurs, the first one will be ignored as well.

Loxone Einbindung der Termine der Stadtreinigung Hamburg

A quick Loxone Tip for People in Hamburg, Germany. That’s why it is in German…

 

Etwas ganz spezielles für die Hamburger Loxone Miniserver Nutzer: ein PicoC Programm, welches es erlaubt mit seiner Anschrift (Straße und Hausnummer) von der Stadtreinigung Hamburg die Termine der nächsten Abholungen einzubinden. Hamburg hat teilweise recht variable Termine und die sind auch nur ein paar Monate im Voraus als Kalender zu importieren. Mit dem Programm kann man folgendes erhalten:

In wieviel Tagen wird das nächste Mal folgendes abgeholt:
– Restmüll
– Papier
– Wertstoffe
– Bio
– Laub (nur im Herbst)
– Weihnachtsbäume (ja, haben wir wirklich – Termine gibt es nur im Januar)

Zudem gibt es Tage mit mehreren Abholungen (Restmüll und Papier oder Restmüll und Wertstoffe). Deswegen erzeugt das Programm auch einen handlichen Statustext für welche Tonnen und an welchem Tag (nächster Termin) bzw. einen Text wie “In 7 Tagen: Restmüll, Papier”.

Der Sourcecode ist einfach ins Programm Bauteil zu kopieren und STRASSE, sowie 999 als Hausnummer durch die korrekte Angabe zu ersetzen. Ursprünglich hat das Programm die Daten von den Texteingängen TI1 und TI2 abgefragt, aber es gibt ja anscheinend keine einfache Text-Variable dafür – also ging es in Programm, die Adresse ändert sich ja nicht ständig…

https://gist.github.com/sarnau/41bde7e9800dad0c40b540114488e161

Mobile Alerts Sensors

ELV is selling affordable weather sensors http://www.elv.de/ip-wettersensoren-system.html, which can be monitored via an iOS application. To allow that all data is transmitted via a gateway to a server, which the application can access. Sadly the protocol is not documented. I don’t think anybody has documented the way the URL has been constructed, especially the MD5 hash over the first part of it.

This sample code integrates sensors into fhem and I wrote it for Mac OS X, but it should work on Linux, too.

All information (and more) is available on GitHub as well.

To integrate the sensors into fhem, add the sensors to the iOS application and check if they show up. You need to note all sensor IDs. The ID is 6 bytes long and represented as a hexadecimal number. This same code supports sensors of type 2 (temperature only) and type 3 (temperature and humidity), which can be recognized via the first two characters of the ID.

You have to adjust the following things in the code:

  • sensors = … Here you have to add all sensors IDs as strings
  • downloadPath This needs to be adjusted to point to the log folder of fhem.
  • vendorid You might want to change the UUID to a new one (just run uuidgen from the terminal)

Python Code

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import sys
import os
import json
import pprint
import datetime
import hashlib
import time

downloadPath = os.path.expanduser('~/fhem-5.7/log')
pp = pprint.PrettyPrinter()

sensors =  set(['021122334455',...,'031122334455'])

def parseJSON(jj):
    if not jj['success']:
        print '#%d %s' % (jj['errorcode'],jj['errormessage'])
        return
    for device in jj['result']['devices']:
        #pp.pprint(device)
        deviceTypeId = device['devicetypeid']
        deviceName = device['name']
        print '%s %s [%d] #%d %s' % (device['deviceid'], deviceName, device['lowbattery'],len(device['measurements']),datetime.datetime.fromtimestamp(device['lastseen']).strftime('%Y-%m-%d %H:%M:%S'))
        year = -1
        month = -1
        logPath = ''
        for measurement in device['measurements']:
            idx = measurement['idx']
            transmitTime = datetime.datetime.fromtimestamp(measurement['c'])
            measurementTime = datetime.datetime.fromtimestamp(measurement['ts'])
            tx = measurement['tx']
            mm = measurement
            del mm['idx']
            del mm['c']
            del mm['ts']
            del mm['tx']

            if year != measurementTime.year or month != measurementTime.month:
                if len(logPath):
                    f = open(logPath, "w")
                    for d in sorted(data):
                        f.write('%s %s\n' % (d,data[d]))
                year = measurementTime.year
                month = measurementTime.month
                logPath = downloadPath + '/S_SENSOR_%s-%d-%02d.log' % (deviceName.upper().replace(' ','_'),year,month)
                data = {}
                try:
                    for line in open(logPath).readlines():
                        vals = line[:-1].split(' ')
                        data[' '.join(vals[:-1])] = vals[-1]
                except:
                    pass
            dataKey = measurementTime.strftime('%Y-%m-%d_%H:%M:%S') + ' SENSOR_' + deviceName.upper().replace(' ','_') + ' '
            if deviceTypeId == 2:
                data[dataKey + 'TEMP:'] = '%.1f' % (measurement['t1'])
            elif deviceTypeId == 3:
                data[dataKey + 'TEMP:'] = '%.1f' % (measurement['t1'])
                data[dataKey + 'FEUCHTIGKEIT:'] = '%.1f' % (measurement['h'])
            else:
                pp.pprint(mm)
        if len(logPath):
            f = open(logPath, "w")
            for d in sorted(data):
                f.write('%s %s\n' % (d,data[d]))


import urllib
import urllib2

devicetoken = 'empty'               # defaults to "empty"
vendorid = '1FB220C4-CC15-4195-97CF-8BE4FD3DAE72'   # iOS vendor UUID (returned by iOS, any UUID will do). Launch uuidgen from the terminal to generate a fresh one.
phoneid = 'Unknown'                 # Phone ID - probably generated by the server based on the vendorid (this string can be "Unknown" and it still works)
version = '1.21'                    # Info.plist CFBundleShortVersionString
build = '248'                       # Info.plist CFBundleVersion
executable = 'Mobile Alerts'        # Info.plist CFBundleExecutable
bundle = 'de.synertronixx.remotemonitor'    # [[NSBundle mainBundle] bundleIdentifier]
lang = 'en'                         # preferred language

request = "devicetoken=%s&vendorid=%s&phoneid=%s&version=%s&build=%s&executable=%s&bundle=%s&lang=%s" % (devicetoken,vendorid,phoneid,version,build,executable,bundle,lang)
request += '&timezoneoffset=%d' % 60        # local offset to UTC time
request += '&timeampm=%s' % ('true')        # 12h vs 24h clock
request += '&usecelsius=%s' % ('true')      # Celcius vs Fahrenheit
request += '&usemm=%s' % ('true')           # mm va in
request += '&speedunit=%d' % 0              # wind speed (0: m/s, 1: km/h, 2: mph, 3: kn)
request += '&timestamp=%s' % datetime.datetime.utcnow().strftime("%s")  # current UTC timestamp

requestMD5 = request + 'asdfaldfjadflxgeteeiorut0ß8vfdft34503580'  # SALT for the MD5
requestMD5 = requestMD5.replace('-','')
requestMD5 = requestMD5.replace(',','')
requestMD5 = requestMD5.replace('.','')
requestMD5 = requestMD5.lower()

m = hashlib.md5()
m.update(requestMD5)
hexdig = m.hexdigest()

request += '&requesttoken=%s' % hexdig

request += '&deviceids=%s' % ','.join(sensors)
#request += '&measurementfroms=%s' % ('0,' * len(sensors))
#request += '&measurementcounts=%s' % ('50,' * len(sensors))

http_header = {
                "User-Agent" : "remotemonitor/248 CFNetwork/758.2.8 Darwin/15.0.0",
                "Accept-Language" : "en-us",
                "Content-Type": "application/x-www-form-urlencoded; charset=utf-8",
                "Host" : "www.data199.com:8080",
                }

# create an urllib2 opener()
opener = urllib2.build_opener()

# create your HTTP request
req = urllib2.Request('http://www.data199.com:8080/api/v1/dashboard', request, http_header)

# submit your request
while True:
    res = opener.open(req)
    parseJSON(json.loads(res.read()))
    print '-' * 40
    time.sleep(10*60)   # wait for 10 minutes

Inside information of the Pronto IR Remote

The Philips Pronto is a programmable remote control. It looks a bit like a Palm Pilot, but it is the most flexible remote control on the market (below the $1000…). The interesting thing with the Pronto is, that it is fully customizable with the Pronto Edit software.

Unfortuanlly the Pronto Edit software is for Windows only, so i reverse engineered the format of the CCF files and also tried to get information about the communication with the Pronto.

CCF file format

Source code for parsing a CCF file:

  • Parse.cp (last change: 10. Nov. 1999; now with support for hidden panels, timers and beeps)
  • ParseGCC.cp (last change: 12. Jul. 2001, source code for the GCC on Linux systems)

Serial Communication

Pronto has a serial port for the communication with a computer. This port works at 115200 baud, 8N1. The communication is pretty simple:

First: get attention of the Pronto. This is done by sending an ACK (0x06) to the Pronto. The Pronto should send a ‘!’ (standby mode) or a ‘*’ (awaiting data, done by holding the left and right key down, while the serial cable was plugged in). If the Pronto don’t respond, try again until timeout.

When the Pronto awaits a command, you can send strings (terminated with CR (0x0D)) like:

irlearn 3500

The Pronto now awaits an infra red signal to learn. The result is send via YModem CRC protocol.

q ccf

Inquiry. The Pronto sends a string containing 4 decimal numbers. The first is the dirty flag (CCF data in the Pronto changed by the user and is not uploaded to the computer yet), the second is the size of the CCF file, the third the the date (in a 8 digit decimal BCD format), the fourth is the time (in a 6 digit decimal BCD format) of the last change of the CCF file.

ul ccf

Uploading a CCF file from the Pronto to the computer. The CCF file is simply send via YModem CRC protocol to the computer.

dl ccf

Downloading a CCF file to the Pronto from the computer. The CCF file is simply send via YModem CRC protocol to the Pronto.

reboot

Reboot the Pronto.

Some links to interesting Pronto websites