Expose TaskReport to outside.

[legiangthanh]

Hi team, We use embulk for importing and exporting data in our company. In our use case, we want to provide and store some statistical information about the task such as the number of records processed, the number of records exported, etc ... but we meet some below the limitation.

1. We can’t get data from TaskReport of input and output plugins after the task finishes.
2. We can’t create a TaskReport for a parser and formatter plugin

Therefore, We want to propose a change that adds the TaskReport to ExecutionResult and adds a new SPI also a change in FileInputPlugin and FileOutputPlugin to create and export TaskReport for a parser and formatter. You can look at an example change at this link legiangthanh/embulk#3.

Thank you in advance for your suggestion and comment.

[dmikurube]

Thanks for your idea. The basic ideas sound good to me, but let me point some nitpicks.

1. How do Embulk users get the "exposed" TaskReport? Your example seems to be implementing getInputTaskReports and getOutputTaskReports, but no one calling them.

2. Would your proposal include parser and formatter? Your example seems to be including only parser. Formatter to be added? How about decoder, encoder, and filter?

3. Is the new method name runThenReturnTaskReport fixed in your mind? That's easy to understand, but maybe too redundant as a standard SPI. We may want to think about a new name.

4. Does just taskReport.merge() work? How would the final getInput/OutputTaskReports be structured in case multiple parsers work for multiple files?

[dmikurube]

@legiangthanh Sorry to be late. Let me leave some responses/questions.

Currently, Plugins will be responsible for this structure. the above example, I got from our implementation. In this proposal we want to have a place to store TaskReport for every plugin, so we don’t force a common structure for each kind of plugin.

Not "Currently", but it's yet your proposal, right? :)

FileInputRunner will take responsibility to combine TaskReport of FileInputPlugin, Decoder, and Parser into finally TaskReport. The structure of the final Report will be


{
	//Key, Value from FileInputPlugin Task Report
	“parser”: {//TaskReport from Parser} 
	“decoders: [//TaskReport from all decoder]
}

E.g.

{
	fileName: “abc”
	…
	“parser”: {"file_path":"/Users/thanh.le/Work/itsm-5316-embulk/./test_1.csv","successes":1,"warnings":0,"warnings_detail”:{}}
	“decoders”: [ {numBytes: 1000}]
}

Q1. For letting a parser generate {"file_path":"/Users/thanh.le/.../test_1.csv", ...}, the parser needs to be aware of file names. But as of now, parsers are ignorant about file names. How do you fill file_path?

Q2. No per-file data from decoders?

Q3. I guesstimate that warnings_detail is a kind of detailed error/warning messages for each problem. But, that could be too large? Is this really the right place for warnings_detail?

[legiangthanh]

@dmikurube

Not "Currently", but it's yet your proposal, right? :)

In my proposal, I provide data as an example to illustrate how we store data for each kind of plugin and combine them, please ignore details data for this discussion.

Q1. For letting a parser generate {"file_path":"/Users/thanh.le/.../test_1.csv", ...}, the parser needs to be aware of file names. But as of now, parsers are ignorant about file names. How do you fill file_path?

In our experiment implementation, We call method hintOfCurrentInputFileNameForLogging to get fileName and store it to parser's TaskReport

Q2. No per-file data from decoders?

It's just an example of data.

Q3. I guesstimate that warnings_detail is a kind of detailed error/warning messages for each problem. But, that could be too large? Is this really the right place for warnings_detail?

Yes, In first our experiment implementation, we try to provide warning details, but it is not feasible to store them in parser's TaskReport

[dmikurube]

In our experiment implementation, We call method hintOfCurrentInputFileNameForLogging to get fileName and store it to parser's TaskReport

That is not an intended use of hintOfCurrentInputFileNameForLogging. Note that we cannot accept that usage in our official examples and standard plugins.

Yes, In first our experiment implementation, we try to provide warning details, but it is not feasible to store them in parser's TaskReport

Yeah, that's expected. That's not the right usage of TaskReport.

[legiangthanh]

That is not an intended use of hintOfCurrentInputFileNameForLogging. Note that we cannot accept that usage in our official examples and standard plugins.

I got it. It is our experiment and doesn't include in this proposal.

[dmikurube]

Okay, then please get things explained deductively by texts (expectedly in English), not inductively by example code and/or data, especially just your "internal" examples. You explained things mainly by examples, but at the same time, "ignore them as they are just internal examples". Then how can we get them understood?