OCTET_STRING encoding problem

[nprobert]

I'm using the SAE J2735-2016 .ASN file. I'm trying to encode an RTCM message (decode works), but it's failing on the OCTET_STRING_uper.c (line 268). As you can see there's a really big number in this debug message which doesn't look right:

Encoding RTCMmessage into 17179869185 units of 8 bits (1..1023, effective 10) (OCTET_STRING_uper.c:252)
Failed to encode element RTCMmessage (OCTET_STRING_uper.c:268)
Failed to encode element RTCMmessageList (constr_SEQUENCE_OF_uper.c:82)
Failed to encode element value (OPEN_TYPE_uper.c:114)
J2735 Encoded: out=-1
Freeing RTCMmessage as OCTET STRING (/home/neal/ProbeStar/GitHub/libj2735v2016/src/OCTET_STRING.c:102)

The C code is here:

#include "MessageFrame.h"

static int msg_count = 1;

int j2735_encode(const char *raw, int size, char *buf, int max)
{
  MessageFrame_t  frame;
  memset(&frame, 0, sizeof(frame));

  RTCMcorrections_t *rtcm = &frame.value.choice.RTCMcorrections;

  A_SEQUENCE_OF(RTCMmessageList_t) list;
  memset(&list, 0, sizeof(list));

  if (!raw || !buf)
    return 0;
  
  // clip
  if (size > 1023)
    size = 1023;

  printf("J2735 Encoded: in=%d, max=%d\n", size, max);

  // build message
  RTCMmessage_t *oct = OCTET_STRING_new_fromBuf(&asn_DEF_RTCMmessage, raw, size);
  ASN_SEQUENCE_ADD(&list, oct);

  // build correction
  rtcm->msgCnt = msg_count++;
  if (msg_count>=128)
    msg_count = 1;
  rtcm->rev = RTCM_Revision_rtcmRev2;
  rtcm->timeStamp = NULL;
  rtcm->anchorPoint = NULL;
  rtcm->rtcmHeader = NULL;
  rtcm->regional = NULL;
  ASN_SEQUENCE_ADD(&rtcm->msgs, &list);

  // build frame
  frame.messageId = 28;   // RTCM
  frame.value.present = MessageFrame__value_PR_RTCMcorrections;

  // encode
  asn_enc_rval_t rc;
  rc = uper_encode_to_buffer(&asn_DEF_MessageFrame, NULL, &frame, buf, max);

  printf("J2735 Encoded: out=%ld\n", rc.encoded);

  free(oct);

  // check
  if (rc.encoded == -1)
    return -1;

  return rc.encoded;
}

[mouse07410]

Yes, UPER (and OER) encoding seems to have some problems.

Would you mind uploading the simplest/shortest ASN.1 file and XER-encoded data that manifest this problem?

[nprobert]

All those mallocs make me nervous. I ran valgrind on the check programs, got nothing. So I wrote a test program to call my routine above and ran valgrind on it.

==1352920== HEAP SUMMARY:
==1352920==     in use at exit: 832 bytes in 24 blocks
==1352920==   total heap usage: 49 allocs, 25 frees, 6,599 bytes allocated
==1352920== 
==1352920== 256 bytes in 8 blocks are definitely lost in loss record 1 of 3
==1352920==    at 0x483B723: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1352920==    by 0x483E017: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1352920==    by 0x14C048: asn_set_add (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920==    by 0x137A82: j2735_encode (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920==    by 0x13765A: rsu_send (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920==    by 0x137539: main (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920== 
==1352920== 576 (256 direct, 320 indirect) bytes in 8 blocks are definitely lost in loss record 3 of 3
==1352920==    at 0x483B723: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1352920==    by 0x483E017: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1352920==    by 0x14C048: asn_set_add (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920==    by 0x1379E9: j2735_encode (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920==    by 0x13765A: rsu_send (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920==    by 0x137539: main (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1352920== 
==1352920== LEAK SUMMARY:
==1352920==    definitely lost: 512 bytes in 16 blocks
==1352920==    indirectly lost: 320 bytes in 8 blocks
==1352920==      possibly lost: 0 bytes in 0 blocks
==1352920==    still reachable: 0 bytes in 0 blocks
==1352920==         suppressed: 0 bytes in 0 blocks
==1352920== 
==1352920== For lists of detected and suppressed errors, rerun with: -s
==1352920== ERROR SUMMARY: 98 errors from 6 contexts (suppressed: 0 from 0)

[nprobert]

I rewrote the code a bit better, I hope. The API is barely documented here. A simpler valgrind output still shows problems, but within OCTET_STRING_new_fromBuf().

int j2735_encode(const char *raw, int size, char *buf, int max)
{
  // assert
  if (!raw || !buf)
    return -1;

  MessageFrame_t  *frame = (MessageFrame_t *)calloc(sizeof(MessageFrame_t), 1);
  // test
  if (!frame) {
    return -1;
  }
  RTCMcorrections_t *rtcm = &frame->value.choice.RTCMcorrections;
    
  // clip
  if (size > 1023)
    size = 1023;

  printf("J2735 Encoded: in=%d, max=%d\n", size, max);

  // build frame
  frame->messageId = 28;   // RTCM
  frame->value.present = MessageFrame__value_PR_RTCMcorrections;

  // build message
  RTCMmessageList_t *list = (RTCMmessageList_t *)calloc(sizeof(RTCMmessageList_t), 1);
  if (!list) {
    free(frame);
    return -1;
  }
  RTCMmessage_t *oct = OCTET_STRING_new_fromBuf(&asn_DEF_RTCMmessage, raw, size);
  RTCMmessage_t *oct = (RTCMmessage_t *)calloc(sizeof(RTCMmessage_t), 1);
  if (!oct) {
    free(list);
    free(frame);
    return -1;
  }
  ASN_SEQUENCE_ADD(&list->list, oct);

  // build correction
  rtcm->msgCnt = msg_count++;
  if (msg_count>=128)
    msg_count = 1;
  rtcm->rev = RTCM_Revision_rtcmRev2;
  rtcm->timeStamp = NULL;
  rtcm->anchorPoint = NULL;
  rtcm->rtcmHeader = NULL;
  rtcm->regional = NULL;
  ASN_SEQUENCE_ADD(&rtcm->msgs, list);

  // encode
  asn_enc_rval_t rc;
  rc = uper_encode_to_buffer(&asn_DEF_MessageFrame, NULL, &frame, buf, max);

  printf("J2735 Encoded: out=%ld\n", rc.encoded);

  ASN_STRUCT_FREE(asn_DEF_MessageFrame, frame);

  // check
  if (rc.encoded == -1)
    return -1;

  return rc.encoded;
}

==1397905== Memcheck, a memory error detector
==1397905== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1397905== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==1397905== Command: ./test_encode
==1397905== 
J2735 Encoded: in=128, max=1398
J2735 Encoded: out=-1
Encoded (128) = -1
==1397905== 
==1397905== HEAP SUMMARY:
==1397905==     in use at exit: 169 bytes in 2 blocks
==1397905==   total heap usage: 8 allocs, 6 frees, 1,809 bytes allocated
==1397905== 
==1397905== 169 (40 direct, 129 indirect) bytes in 1 blocks are definitely lost in loss record 2 of 2
==1397905==    at 0x483DD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1397905==    by 0x13823E: OCTET_STRING_new_fromBuf (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1397905==    by 0x137815: j2735_encode (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1397905==    by 0x137632: rsu_send (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1397905==    by 0x137523: main (in /home/neal/NTCNA/CInfrastructure/Software/ntripclient/test_encode)
==1397905== 
==1397905== LEAK SUMMARY:
==1397905==    definitely lost: 40 bytes in 1 blocks
==1397905==    indirectly lost: 129 bytes in 1 blocks
==1397905==      possibly lost: 0 bytes in 0 blocks
==1397905==    still reachable: 0 bytes in 0 blocks
==1397905==         suppressed: 0 bytes in 0 blocks
==1397905== 
==1397905== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

[nprobert]

Added ASN_STRUCT_FREE(asn_DEF_OCTET_STRING, oct); near the bottom and got no valgrind errors which kinda tells me that the OCTET_STRING didn't get added to the RTCMmessageList_t structure by ASN_SEQUENCE_ADD somehow.

Traced this back to the asn_set_add() function. Added some debugging code in there and it started working. Removed it and it failed again. Wtf!? Either way, the test message still doesn't get encoded.

[liubing]

calloc(sizeof(RTCMmessageList_t), 1) should be
calloc(1, sizeof(RTCMmessageList_t))

[mouse07410]

I'm sorry, I don't have a clear idea of what you're doing, and SAE J2735-2016 "at large" is way too huge for me to comprehend or play with.

I say again - would you be able to create and upload a simple reproducer of the problem? That would include a comprehensible small ASN.1 file and an XER example of encoded data?

[nprobert]

J2735 really challenges this compiler, breaks it bad probably. Seems it can't even encode an constrained INTEGER (long) in UPER. The value is supposed to be 12, but looks like a pointer is f'd up in INTEGER_uper.c.

About to encode DSRCmsgID (/home/neal/ProbeStar/GitHub/libj2735v2016/src/constr_SEQUENCE_uper.c:379)
Encoding MessageFrame->messageId:DSRCmsgID (/home/neal/ProbeStar/GitHub/libj2735v2016/src/constr_SEQUENCE_uper.c:402)
Encoding NativeInteger DSRCmsgID 78202640 (UPER) (/home/neal/ProbeStar/GitHub/libj2735v2016/src/NativeInteger_uper.c:60)
Value 78202640 (04/4) lb 0 ub 32767 ext (/home/neal/ProbeStar/GitHub/libj2735v2016/src/INTEGER_uper.c:178)
Failed to encode element DSRCmsgID (/home/neal/ProbeStar/GitHub/libj2735v2016/src/INTEGER_uper.c:188)

[nprobert]

Memory allocation is badly broken in ways not even valgrind can find. So I just built up my J2735 message by avoiding any dynamic memory allocation. And voila, things work now.

//
// J2735 Encode
//

#include "MessageFrame.h"

static int msg_count = 1;

int j2735_encode(const char *raw, int size, char *buf, int max)
{
  // assert
  if (!raw || !buf)
    return -1;

  // clip
  if (size > 1023)
    size = 1023;

  // storage
  MessageFrame_t  frame;
  memset(&frame, 0, sizeof(frame));
  RTCMmessageList_t list;
  memset(&list, 0, sizeof(list));
  RTCMmessage_t     msg;
  memset(&msg, 0, sizeof(msg));
  
  // frame
  frame.messageId = DSRCmsgID_rtcmCorrections_D;   // RTCM
  frame.value.present = MessageFrame__value_PR_RTCMcorrections;

  // corrections
  RTCMcorrections_t *rtcm = &frame.value.choice.RTCMcorrections;
  rtcm->msgCnt = msg_count++;
  if (msg_count>=128)
    msg_count = 1;
  rtcm->rev = RTCM_Revision_rtcmRev2;

  // message list
  rtcm->msgs.list.count = 1;
  rtcm->msgs.list.size = 1;
  
  // message
  msg.buf = (void *)raw;
  msg.size = size;
  RTCMmessage_t *pmsg = &msg;
  rtcm->msgs.list.array = &pmsg;
  list.list.array = &pmsg;
  list.list.count = 1;
  list.list.size = 1;
  
#ifdef CONSTRAINT_CHECKING
  // walk
  char err[200];
  size_t len = sizeof(err);
  if (asn_check_constraints(&asn_DEF_MessageFrame, &frame, err, &len) != 0) {
    fprintf(stderr, "%s\n", err);
    return -1;
  }
#endif

  // encode
  asn_enc_rval_t rc;
  rc = asn_encode_to_buffer(0, ATS_UNALIGNED_CANONICAL_PER, &asn_DEF_MessageFrame, &frame, buf, max);
   
  // check
  if (rc.encoded == -1)
    return -1;
  else if (rc.encoded > max)
    fprintf(stderr, "Encoding buffer is too small\n");

  return rc.encoded;
}

[mouse07410]

Encoding RTCMmessage into 17179869185 units of 8 bits (1..1023, effective 10) (OCTET_STRING_uper.c:252)

@nprobert this looks like OCTET_STRING_uper.c code is getting a pointer that it mis-interprets as an integer - a number of units. That could also explain at least one memory leak. Would you be able to complete the trace and find who's passing a pointer in place of a value here?

[nprobert]

I agree. The compiler should catch this, an evil cast, but it doesn't. I'm guessing this was an indirect function call through a table. Or a mis-cast void pointer.