Documenting "inspect" and context awareness udf.rst

[daviddkovacs]

Hi, I made a quick description on how to use the "inspect" function to print within the UDFs.

moreover, I tried to give a brief explanation on how users can provide their own local side variables to be used within the UDF.

[jdries]

Nice, thanks!
For the bit about inspecting variables, we may want to integrate it with the existing explanation about logging?
https://open-eo.github.io/openeo-python-client/udf.html#logging-from-a-udf
Or if somehow the existing explanation was hard to find, suggestions for improvement are welcome.

[soxofaan]

Hi, thanks for taking the time to contribute!

About the part on inspect in UDF, I agree that it might be better to integrate that with the existing inspect docs at https://open-eo.github.io/openeo-python-client/udf.html#logging-from-a-udf or make them easier to discover (e.g. move it up or sprinkle some references around).

About the part on passing through the context: that is indeed a valid topic to document better. Also see the discussion at #520
A thing that should be added to the current PR is how to pass the context to the "parent" process of the UDF

[daviddkovacs]

Hi, thanks for taking the time to contribute!

About the part on inspect in UDF, I agree that it might be better to integrate that with the existing inspect docs at https://open-eo.github.io/openeo-python-client/udf.html#logging-from-a-udf or make them easier to discover (e.g. move it up or sprinkle some references around).

About the part on passing through the context: that is indeed a valid topic to document better. Also see the discussion at #520 A thing that should be added to the current PR is how to pass the context to the "parent" process of the UDF

Yes, indeed it is better to integrate into the existing docs.
I am looking into passing the context ot the parent process of the UDF (i.e. the part NOT in apply_datacube function right ?), however I didnt manage to do that, nor found anything on it. Can you indicate me where it is shown, and I'll integrate it in the PR

[soxofaan]

good point, it's not easy to find succint examples on properly using context in UDF.

We have some unit test coverage in the VITO backend on this for example at https://github.com/Open-EO/openeo-geopyspark-driver/blob/cdd731ce6d684eba894beff7c8ac78266ddf12b0/tests/test_api_result.py#L718-L889, but that's probably a bit cryptic.

I think there are two use cases to document:

1. directly passing context to run_udf

udf = openeo.UDF(
    "...", 
    context={"factor": 12.34},
)
cube = cube.apply(udf)

2. passing context to parent process, and pass it through in run_udf

udf = openeo.UDF(
    "...", 
    context={"from_parameter": "context"},
)
cube = cube.apply(udf, context={"factor": 12.34})

Both of these patterns have their usefulness .The first is simpler to reason about. The second is the approach to take when the context comes from "higher up", e.g. UDP parameters

[soxofaan]

That being said, I think the python client should make it simpler to get that second usage pattern right. I made a ticket for that:

• automatically pass through context to UDF from parent process #618

[daviddkovacs]

Added proper description, shorter lines and passing "context" to parent udf

[soxofaan]

some more notes

[soxofaan]

This section is still redundant given the existing "Logging from a UDF" section at

openeo-python-client/docs/udf.rst

Lines 616 to 645 in 21922f1

	Logging from a UDF
	=====================
	From time to time, when things are not working as expected,
	you may want to log some additional debug information from your UDF, inspect the data that is being processed,
	or log warnings.
	This can be done using the :py:class:`~openeo.udf.debug.inspect()` function.
	For example: to discover the shape of the data cube chunk that you receive in your UDF function:
	.. code-block:: python
	:caption: Sample UDF code with ``inspect()`` logging
	:emphasize-lines: 1, 5
	from openeo.udf import inspect
	import xarray
	def apply_datacube(cube: xarray.DataArray, context: dict) -> xarray.DataArray:
	inspect(data=[cube.shape], message="UDF logging shape of my cube")
	cube.values = 0.0001 * cube.values
	return cube
	After the batch job is finished (or failed), you can find this information in the logs of the batch job.
	For example (as explained at :ref:`batch-job-logs`),
	use :py:class:`BatchJob.logs() <openeo.rest.job.BatchJob.logs>` in a Jupyter notebook session
	to retrieve and filter the logs interactively:
	.. image:: _static/images/udf/logging_arrayshape.png
	Which reveals in this example a chunking shape of ``[3, 256, 256]``.

I'd propose to finetune the existing docs if that is necessary

[soxofaan]

used throughout the user side of script

I'm not sure what you mean here, e.g. with "user side". And what "script" are you referring to? The script that build the process graph, or the UDF script?

[daviddkovacs]

from the user side, what I mean is the script where the user runs the UDF.

[soxofaan]

I'm still a bit confused what you mean:

where the user runs the UDF.

the user does not run the UDF user side, it's the backend that executes the UDF backend-side

[daviddkovacs]

Sorry, the script where the user defines the UDF script.

[soxofaan]

Suggested change

	Once, these variables are defined within `context` dictionary, the UDF needs to be made context aware, by adding `context={"from_parameter": "context"}` at the end of your UDF.
	Once these variables are defined within `context` dictionary, the UDF needs to be made context aware, by adding `context={"from_parameter": "context"}` at the end of your UDF.

variables are defined within context dictionary

This is a bit confusing, because in your example you define them in a user_variable dictionary

[soxofaan]

FYI: The UDF is not the parent here, apply is the parent.

the hierarchical flow is apply -> run_udf -> your UDF