Problem getting sum to work in CALI_REPORT_CONFIG define

[dnystrom1]

Below is a section of a bash script that I am using to collect uncore memory counters on a KNL
node. I am collecting counter data for both the DDR memory controllers and the MCDRAM
memory controllers. To do this, it appears that only a single MPI process can be used for each
uncore counter. There are 14 of these counters for the "read" case and 14 for the "write" case.
I am using separate runs to collect the "read" counters and then the "write" counters. The
following section of script works but note that I have the line defining CALI_REPORT_CONFIG
commented out. For this particular case, I am running with an MPI rank per hardware thread
for a total of 272 ranks and I am using the first 14 ranks to collect the counter data. When I
collect my counter data, I have instrumented my application to collect data for a section of
my application labeled "center_p" and for a section of data labeled "test_advance_p". The
section labeled "center_p" is only collected once per run and does not need to be summed.
The section labeled "test_advance_p" is collected once per timestep or 100 times for my
current test case. Thus, for test_advance_p, it is very useful to sum that data so I only have
one result per counter per run.

If I uncomment the line which defines CALI_REPORT_CONFIG, my runs fail to collect any data,
it seems, and I get the following error messages in my log file.

== CALIPER: (1): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (2): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (3): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (5): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (4): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (6): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (7): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (8): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (9): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (10): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (11): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (12): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (13): default: Report: config parse error: Invalid number of arguments for sum
== CALIPER: (0): default: Report: config parse error: Invalid number of arguments for sum

#---------------------------------------------------------------------------------------
elif [ $NUM_RANKS -gt "1" ]
then
if echo $METRIC_GROUP | grep -q "rd"
then
if [ $PMI_RANK = 0 ]
then
METRIC_RANK="knl_unc_imc0::UNC_M_CAS_COUNT:RD:cpu=0"
elif [ $PMI_RANK = 1 ]
then
METRIC_RANK="knl_unc_imc1::UNC_M_CAS_COUNT:RD:cpu=0"
elif [ $PMI_RANK = 2 ]
then
METRIC_RANK="knl_unc_imc2::UNC_M_CAS_COUNT:RD:cpu=0"
elif [ $PMI_RANK = 3 ]
then
METRIC_RANK="knl_unc_imc3::UNC_M_CAS_COUNT:RD:cpu=0"
elif [ $PMI_RANK = 4 ]
then
METRIC_RANK="knl_unc_imc4::UNC_M_CAS_COUNT:RD:cpu=0"
elif [ $PMI_RANK = 5 ]
then
METRIC_RANK="knl_unc_imc5::UNC_M_CAS_COUNT:RD:cpu=0"
elif [ $PMI_RANK = 6 ]
then
METRIC_RANK="knl_unc_edc_eclk0::UNC_E_RPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 7 ]
then
METRIC_RANK="knl_unc_edc_eclk1::UNC_E_RPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 8 ]
then
METRIC_RANK="knl_unc_edc_eclk2::UNC_E_RPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 9 ]
then
METRIC_RANK="knl_unc_edc_eclk3::UNC_E_RPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 10 ]
then
METRIC_RANK="knl_unc_edc_eclk4::UNC_E_RPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 11 ]
then
METRIC_RANK="knl_unc_edc_eclk5::UNC_E_RPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 12 ]
then
METRIC_RANK="knl_unc_edc_eclk6::UNC_E_RPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 13 ]
then
METRIC_RANK="knl_unc_edc_eclk7::UNC_E_RPQ_INSERTS:cpu=0"
fi
else
if [ $PMI_RANK = 0 ]
then
METRIC_RANK="knl_unc_imc0::UNC_M_CAS_COUNT:WR:cpu=0"
elif [ $PMI_RANK = 1 ]
then
METRIC_RANK="knl_unc_imc1::UNC_M_CAS_COUNT:WR:cpu=0"
elif [ $PMI_RANK = 2 ]
then
METRIC_RANK="knl_unc_imc2::UNC_M_CAS_COUNT:WR:cpu=0"
elif [ $PMI_RANK = 3 ]
then
METRIC_RANK="knl_unc_imc3::UNC_M_CAS_COUNT:WR:cpu=0"
elif [ $PMI_RANK = 4 ]
then
METRIC_RANK="knl_unc_imc4::UNC_M_CAS_COUNT:WR:cpu=0"
elif [ $PMI_RANK = 5 ]
then
METRIC_RANK="knl_unc_imc5::UNC_M_CAS_COUNT:WR:cpu=0"
elif [ $PMI_RANK = 6 ]
then
METRIC_RANK="knl_unc_edc_eclk0::UNC_E_WPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 7 ]
then
METRIC_RANK="knl_unc_edc_eclk1::UNC_E_WPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 8 ]
then
METRIC_RANK="knl_unc_edc_eclk2::UNC_E_WPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 9 ]
then
METRIC_RANK="knl_unc_edc_eclk3::UNC_E_WPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 10 ]
then
METRIC_RANK="knl_unc_edc_eclk4::UNC_E_WPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 11 ]
then
METRIC_RANK="knl_unc_edc_eclk5::UNC_E_WPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 12 ]
then
METRIC_RANK="knl_unc_edc_eclk6::UNC_E_WPQ_INSERTS:cpu=0"
elif [ $PMI_RANK = 13 ]
then
METRIC_RANK="knl_unc_edc_eclk7::UNC_E_WPQ_INSERTS:cpu=0"
fi
fi

    if [ $PMI_RANK -lt $NUM_RANKS ]
    then
        export CALI_SERVICES_ENABLE=trace,event,papi,report
        export CALI_PAPI_COUNTERS=${METRIC_RANK}
       #export CALI_REPORT_CONFIG="SELECT *,sum(papi.${METRIC_RANK}) GROUP BY prop:nested FORMAT json"
        export CALI_REPORT_FILENAME=cali_int_knl_68_nppn_272_int/cali_${METRIC_RANK}.json
    fi

    $MY_APP

#---------------------------------------------------------------------------------------

For the case where the line which defines CALI_REPORT_CONFIG is comment out, data collection appears
to be successful and to produce reasonable results. Here is an example of the data from the .json output
file.

% cat cali_knl_unc_edc_eclk0::UNC_E_RPQ_INSERTS:cpu=0.json
event.begin#annotation papi.knl_unc_edc_eclk0::UNC_E_RPQ_INSERTS:cpu=0 event.end#annotation annotation
center_p 163607
4158274626 center_p center_p
test_advance_p 15140278
23603582 test_advance_p test_advance_p
test_advance_p 102831
22010440 test_advance_p test_advance_p
test_advance_p 112297
21548302 test_advance_p test_advance_p
test_advance_p 114400
21477455 test_advance_p test_advance_p
test_advance_p 114415
21316194 test_advance_p test_advance_p
<.... more similar data ....>

Does anyone have any ideas of why I get the errors about an invalid number of arguments for sum?
I am wondering if it might be because of the counter name which has some embedded colons and
an equal sign but perhaps it is something else that I am doing wrong and not understanding.

[dnystrom1]

Hmmm. Looks like the cut and paste of the bash script snippet did not render very well.

[daboehme]

Hi @dnystrom1 ,

Yeah, I think the issue is the '=' character in the metric names. You should be able to escape either the character with a backslash or the whole metric name with quotation marks, e.g. like this:

CALI_REPORT_CONFIG="SELECT *,sum(\"papi.${METRIC_RANK}\") GROUP BY prop:nested FORMAT json"

Let me know if it helps!

[dnystrom1]

Hi @daboehme ,

Thanks for your suggestion above with the escaped double quotes. Indeed, that does work, i.e. your second solution with escaped double quotes around the whole metric name.

Thanks,

Dave

[daboehme]

Great! Glad to know it works.