New configuration system and DAQ recording

[christoph2]

So guys, this (v. 0.22) is a really comprehensive update... 🍾 🍾 🍾

The two most important changes are:

• On multiple requests there is now native support for DAQ-lists!
• New traitlets based configuration system.

Configuration System

To get a first impression, run

xcp-profile create | less

Note

Using xcp-profile is not really required, every pyXCP based program/script can handle profiles.

To write a configuration file, use option -o (-d works the same way). Note: extension .py is required in any case.

It is recommended but not necessary to use pyxcp_conf.py as file name,
because it is the file pyXCP looks for, if there is no -c option.

Simply run

python xcphello.py

instead of

python xcphello.py -c pyxcp_conf.py

Upgrading an existing configuration

Convert a legacy configuration file as follows:

xcp-profile convert -c conf_can_user.toml -o conf_can_user.py

If there's no output option (-d or -o), the result is written to stdout.

---

• The infamous issue 130 is now fixed.

This means, the new configuration system now interprets the parameters

c.Transport.Can.can_id_master = 47
c.Transport.Can.can_id_slave = 11

correctly.

For the sake of compatibility, nothing is changed when using old .JSON/.TOML files, but the IDs are crossed while converting your configuration files.
For the utterly confused, the logger dumps out some information on what's going on, e.g.:

2024-08-06 16:25:54 INFO     XCPonCAN - Interface-Type: 'kvaser' Parameters: [('channel', '0'), ('fd', False), ('bitrate', 500000),                 can.py:333
                             ('receive_own_messages', False), ('sjw', 2), ('tseg1', 5), ('tseg2', 2)]
                    INFO     XCPonCAN - Master-ID (Tx): 0x00000300S -- Slave-ID (Rx): 0x00000301S                                                   can.py:334
                    INFO     XCPonCAN - Using Interface: 'Kvaser U100, S/N 1093742 (#1)'                                                            can.py:256
2024-08-06 16:25:55 INFO     XCPonCAN - Filters used: [{'can_id': 769, 'can_mask': 2047, 'extended': False}]                                        can.py:257
                    INFO     XCPonCAN - State: BusState.ACTIVE

---

Python as configuration language opens completely new posibilities, e.g.: a Python function instead of a .DLL can be used for seed-and-key unlocking:

def SeedKeyXCP(resource: int, seed: bytes):
    temp0 = seed[0]
    temp1 = seed[1]
    temp2 = seed[2]
    temp3 = seed[3]

    temp = (temp3 << 24) | (temp2 << 16) | (temp1 << 8) | temp0
    temp = ((temp >> 5)) | (temp << 23)
    temp = (temp * 7)
    temp = temp ^ 0x26031961

    key = bytearray(9)
    key[0] = ((temp >> 0)   & 0x00FF)
    key[1] = ((temp >> 8)   & 0x00FF)
    key[2] = ((temp >> 16)  & 0x00FF)
    key[3] = ((temp >> 24)  & 0x00FF)
    return bytes(key)

#c.General.seed_n_key_dll = 'SeedNKeyXcp.dll'
c.General.seed_n_key_function = SeedKeyXCP

Tested with Vector's XCPsim (CANape 20.0)

DAQ lists

First of all, there's an example run_daq demonstrating how-to DAQ -- Please read on before trying to run!

run_daq was tested with CANape 20.0 and XCPlite;
The CANape addresses may work, the XCPlite ones will certainly NOT!

DAQ measurements can be done in two ways:

• Online, collect and or process data while DAQ measurement is running.
• Offline, data is recorded to a .xmraw file and then processed afterwards.

This basic descision looks like this:

# daq_parser = DaqToCsv(DAQ_LISTS)  # Record to CSV file(s).
daq_parser = DaqRecorder(DAQ_LISTS, "run_daq", 2)  # Record to ".xmraw" file.

Note: DaqToCsv could be used as as startingpoint for own live processing.

---

If you look at the signature of the callback function

def on_daq_list(self, daq_list: int, ts0: int, ts1: int, payload: list):
	...

or a finished recording

timestamp0,timestamp1,byteCounter,sbyteCounter,wordCounter,dwordCounter
1724389279036840900,1970172536,141, -115, 4749, 70285
1724389279067043600,1985243936,142, -114, 4750, 70286
1724389279128505800,2046627736,146, -110, 4754, 70290
1724389279128540700,2061804936,147, -109, 4755, 70291
1724389279144103000,2077478736,148, -108, 4756, 70292

you notice two timestamps, timestamp0 and timestamp1.
timestamp0 is created by pyXCP, while timestamp1 is the DAQ timestamp from your ECU or simulator (contained in the first ODT).

Setup

There are basically only two data-structures the user has to care about:

from pyxcp.daq_stim import DaqList

A DaqList correspondeds to a XCP DAQ-list and looks conceptually like this:

class DaqList:
	meas_name: str
	event_num: int 
	stim:  bool
	enable_timestamps:  bool
	measurements: list[Measurement]

• meas_name is important when it comes to storage later on, it can be used for example as a name of a .csv file, a SQL table, a MDF channel-group, and so on.
• event_num the number of the event the DAQ-list is attached to, depending of the implementation, events may only used once.
• stim is the direction of the DAQ-list (DAQ/STIM), and must be Falsefor now.
• enable_timestamps timestamp-generation may enabled or disabled or may fixed, this is completely implementation-defined.
• measurements is a list of tuples with the following content:

  • name in most cases the name of the A2L MEASUREMENT element.

  • address

  • address_extension

  • datatype datatype mnemonic, to get an overview run:

     python -c "from pyxcp.recorder import DATA_TYPES; print(DATA_TYPES)"

    Note

    16-bit floating-point variables are fully supported, but the availability is compiler-dependent;
    F16 (5-bit exponent and a 11-bit mantissa), BF16 (8-bit exponent and a 8-bit mantissa, "brain-float")
    for details s. cppreference, and here

The allocation and optimization of ODTs is done automatically by pyXCP.

After a recording is done, you can now start processing; the basic pattern is as follows:

from pyxcp.recorder import XcpLogFileDecoder

class Decoder(XcpLogFileDecoder):

    def __init__(self, recording_file_name: str) -> None:
		"""
		Regular Python constructor, gets the file name of the recording.
		At this point you may set an output file name, e.g.
		
		self.my_file_name = Path(recording_file_name).with_suffix(".txt")		
		"""
		
	def initialize(self) -> None:
		"""
		Here you may create files, objects, ...
		"""
		
	def finalize(self) -> None:
		"""
		Close files, and the like at this stage.
		"""
		
	def on_daq_list(self, daq_list_num: int, timestamp0: int, timestamp1: int, measurements: list) -> None:
		"""
		As you may guess, this is called on every finished DAQ-list.
		"""
		
decoder = Decoder("my_recording.xmraw")
res = decoder.run()	

The are three examples for common use-cases (currently no Excel, sorry):

• ex_arrow create Parquet files.
• ex_mdf create MDF file.
• ex_sqlite create SQLite3 database.

Miscellaneous / Minor changes

• The build system is now poetry -- setup.py is history!
  As an alternative to python setup.py develop one should use pip install -e .

• Timestamping was somewhat chaotic (time.time(), time.perf_counter(), ...) -- now handled by a C++ extension, timestamps are now UTC values (64-bit, with nano-seconds resolution). The startup timestamp (incl. timezone, UTC-offset, and DST-offset) is accessible:

with ap.run() as x:
    print("Start DT: ", x.start_datetime)	

• If you already have a communication link to your bus/network, you may re-use the connection. (s. parallel access to one device on windows #162), e.g.:

import can

from pyxcp.cmdline import ArgumentParser

can_if = can.Bus(interface="kvaser", channel="0", fd=False, bitrate=500000)
ap = ArgumentParser(description="external interface test")

with ap.run(transport_layer_interface=can_if) as x:
	x.connect()	
	x.disconnect()

can_if.shutdown()	# Ownership is (of cause) not transfered to pyXCP.

The type of the interface is currently not deduced, so the interface and the transport-layer configuration must match, e.g.:

c.Transport.layer = 'CAN'

• rich as a terminal-IO library, more comprehensible exceptions and colorful logging.
• STIM is not implemented yet, but there's some infrastructure.
• Also work-in-progress: PTP / IEEE1588 hardware-timestamping support (C++ extension).
• Building on RasbPIs and Beagles is problematic, due to memory restrictions -- Docker to the rescue, but I'll open a separate discussion on this topic.
• Minimum supported Python version is 3.8.

---