Automatic ratio calculations using Reinforcement Learning.

[Unai-HP]

Hi, I'm a university student doing my final degree project, which is adding a Reinforcement Learning ("AI automatic learning") base calculator for carb ratio and insulin sensitivity to xDrip, this is, inputing only carbs and insulin the ratios would be calculated automaticaly.

However, I wanted to ask if there is any type of documentation of the code, since I've being searching for it and found nothing. Is there any class diagram or dependency diagram of the code (I myself have found nothing)? I've tried 6-7 class diagram generators (they use the existing code to create them) but half of them don't work, and the rest have terrible results.

On the other hand, is anyone interested in this feature? How do you think this feature should work?

Sorry for my bad English by the way, not my first language.

[Navid200]

I am an xDrip user and one of the volunteers helping maintain xDrip. Everything I say here is my personal opinion.

I personally see no problem with us supporting your final year project. But, approving and adding your project to xDrip is a different matter.
I have no problem with that either as long as it meets all the requirements.

Just to clarify, xDrip is used worldwide by those suffering from diabetes. We always have to look at the outcome from the angle of how it would impact the user. If a change to xDrip could cause to confuse even one user resulting in them taking more insulin than they should have, we need to avoid that change.
I have seen programmers come here and submit a pull request and when their submission was rejected, they left and never came back.

xDrip is open source. It has contributions from more than 72 different volunteers. These individuals don't work for the same company and don't report to a reporting hierarchy. It is not easy to mandate and enforce standards.

There are developers who open a pull request (PR) adding a new feature. It gets approved and added to xDrip. Later, there is a bug, either because Android is updated or because a phone manufacturer changes the phone firmware or because ... But, the individual is gone. We need to resolve the bug because users complain about it. We have not seen the majority of the 72 contributors I mentioned for years.

Individuals come here and point out that the xDrip code is inconsistent and open a PR. They focus on the code and are surprised if their PR is not approved. Our objective is the app and how it impacts the user. The code is not our goal. It is a tool.
I use xDrip 24/7 because I am diabetic. As a user, I couldn't care less if the code is inconsistent as long as it doesn't tell me that my blood glucose is perfect when I really am low and need sugar. I need it to work correctly.

I have opened a submission after having tested it thoroughly. I have asked other developers to test it. They have. After approval, we have realized that there was a mistake. There is always a chance for mistakes. We cannot just approve submissions without weighing all the risks and asking why we really need it.

I don't mean to discourage you. I just want you to be aware of what xDrip is.
You have done the right thing by opening this discussion.
I will tag others who may have opinions/suggestions for you.
@tzachi-dar @psonnera @jamorham @tolot27 @jwoglom

[Unai-HP]

Thank you for such a fast reply. My idea was to do a pull request to make it upstream, but I don't take the merge for granted, don't worry. I understand that xDrip is a big project and so it has high standards, as I think it should have. I use it 24/7 too since I have diabetes, and so I see it as a (game changing) tool. However, being a Software Engineer I'm forced to use things like class diagrams and so, and at the same time I see them as a great feature to ease future developing. I hope I will be around after my FDP. For example, I want to fix the issue with onboard insulin not showing, which has been bothering me for some time.

Again, thank you for the reply.

[Navid200]

It will be fantastic to have you as a contributor.
I am sorry about the IoB issue. It is resolved already: https://navid200.github.io/xDrip/docs/IoB_not_showing.html

[Unai-HP]

I thought I had the latest update installed, nope. Its now resolved, thanks.

[Navid200]

There are different update channels. I have explained it here: https://navid200.github.io/xDrip/docs/Updates.html

[jamorham]

@Unai-HP You should find all treatment data via the Treatments class and you can use helper static methods like latest(x) to retrieve a list of Treatments which would include insulin and carb entries. You can also retrieve sensor glucose values from the BgReading class using a similar static method latest(x) you can marry these up to produce your model to evaluate sensitivity.

I would recommend creating a new package inside processing and place any new classes there. Make any methods you create maintain their own state and read from the above mentioned classes and provide the output. You can use PersistentStore and FastStore classes to store any operational data or results as needed.

This way your implementation is effectively stand-alone simply processing the available data and then to integrate in to xDrip, for example in the predictive simulation model we can call out to your methods to receive data. In this way we can prototype a dual system to examine results from any processing you make vs the baseline and give users the option to use one or the other or potentially both without creating many modifications to the existing code base and therefore making the whole thing easier to manage and potentially merge at some future point.

Your English is absolutely fine and clear.

[Unai-HP]

Thank you for the help and suggestions, I'll make a draft pull request when I've something functional.

[Unai-HP]

Hi @Navid200 @jamorham again. I've been doing changes in my fork and the next thing I wanted to do was changing the UI to include a simple option like "Use RL". However changing pref_advanced_settings.xml doesn't seem to do anything. Is there anything more I have to do that I've missed? I'm using Android Studio, gradle plugin 3.4.3 and studios 6.8, with java 11, and there are no errors in the gradle build aside from "fatal: No names found, cannot describe anything.", which doesn't seem to stop the installation of the new build.

[Unai-HP]

Thank you so much for the help. Unfortunatily it doesn't work for me. I'll keep searching for a solution.

It seems that it builds the changes (I've removed the settings xml and now it doesn't work), but ignores the string changes especificaly.

Edit: Ok, it seems that in my case I have to delete xdrip in the emulator too, but it works with the clean and rebuild trick. Again, thank you very much, you've saved me like 2-3 days of work :).

Should this be considered as a bug? Searching for workarounds I've seen that it may be a problem with gradle's configuration. I can open an issue if needed.

[jamorham]

I use android studio bumblebee because of having problems with bugs in later versions.

[Unai-HP]

After some 2 weeks writing documentation for my FDP I've made other changes to the code (the xml files) and now the interface is not updated, not even with cleaning and rebuilding. I've tried bumblebee too but it doesn't appear to work either. The only thing changed it's that I've made a marge with upstream. Is this widespread? If not, can any of you give me the exact versions of Android Studio the instructions to build and use the emulator? I'm sorry for asking for help, but yesterday the only thing I did was try to solve this problem and I wasn't not able to.

[Navid200]

After updating to upsteam, have you updated the gradle JDK to 11?

[Unai-HP]

Hi. Yes, I've updated the JDK. It only updates once when I make a clean install of android studio, and build the app. I'm testing changes changing "xdrip_plus_prefs.xml".

After doing a clean install again and wiping other things, for some reason it works now.

[Navid200]

My main concern with any simulator is that it could cause serious problems for someone who may take it as what it is not.

There are many reasons a simulator could be wrong. There are many factors affecting our blood glucose. Insulin and carbs are just 2 of many.
Stress, exercise, menstrual cycle, or Glycemic index are just some examples of items that directly or indirectly affect blood glucose.

Delegating more and more to AI, in this case, is something that could have catastrophic consequences.

When I read you talk about learning, unless I misunderstand, I take that for a while the simulator is going to be even more wrong than usual until it learns. Well, what is the user supposed to do while the ai is learning? How do we make sure everyone understand that?

I am not even happy with the existing simulator because I am worried someone may use it the wrong way. So, please note this all is only my personal opinion.

[Unai-HP]

Yes, I mostly agree with what you are saying. The simulation, if taken literally and without thought can be very dangerous. However, in my opinion it's a great tool to do the carb to insulin calculations, specially if using an insulin pen and not a pump. It's way better than eye bolus (way more dangerous than the calculation) too, which is what I did before using it.

In regard to the factors, I started this project in order to try to improve the simulations "rigidness", since the RL agent (if done right) should take into account the insulin patterns of each user (for example menstrual cycle). In my opinion, things like closed loop systems and use of machine learning are the future of diabetes management. This can be seen in some commercial closed loops systems, for example. This is not the same of course, since it's not a medically approved system and should not ever be used to take a final decision, only as an educational and helper system.

Regarding safety, I agree that before enabling any simulation it should give a clear warning stating that it's only a help with the decision the user itself has to take, not a certainty, and that it could be very dangerous, specially with a lack of data. Any concern with safety has to be taken in consideration. Finally, if the dev team has a consensus that this is too risky and not useful, I can revert to carb/sensitivity idea (I will have to debate it with my teachers, but I think it can be done). I only want to improve the app and help people with it, and if the dev team thinks what I'm doing does not, I would try to correct it as much as I can.

[Unai-HP]

@jamorham @Navid200 Hi, after some time I've mostly finished coding the feature. It has the necessary UI changes to enable it and see the RL prediction in the application status bar. However, I have yet to implement the final version of the RL model, which for now has quite terrible results. I know that a very good model is possible. In the paper "Toward a Fully Automated Artificial Pancreas System Using a Bioinspired Reinforcement Learning Design: In Silico Validation" (https://ieeexplore.ieee.org/document/9115809) they describe a RL model that reaches a 92% time in range (using only bg as input, with a pump).

Knowing this, should I wait until I have the model trained and fully usable to do the pull request, or should I do it now?

[Navid200]

I am not certain about the concept described here.

"should I wait until I have the model trained"

Is this training something that you need to do once before users can use your simulator?

Or, is this training something that every user needs to do before they each can use your simulator?

[Unai-HP]

It has to be trained at least once (and after it can be retrained with user data to adapt to the users needs and patterns. This is outside the scope for now thought). What I'm saying above is that if there is no model to be used, the feature can not be used in a real scenario (although it can be tested with a dummy model, which gives terrible results like giving negative insulin with 160mg). Sorry for not being clearer. I hoping for a decent model before June.

[yosepito]

Hi to everyone. I use Xdrip since several years ago and could share my data to train this model if someone tells me how to download the data.
To me, the relevant issue would be to define three time periods (breakfast, lunch, dinner) and that we have something implemented in Xdrip or in Nightscout sites to provide hints of dynamics of the C/I ratio for each main meal moment, for instance showing dynamics for the latest xxx days/weeks. When I'm ill with fever, I nearly need to double both basal and meal doses for a certain amount of carbohydrates. But I would love to see predictions based on reinforcement learning that make the computations for me.
Is there any progress on this ?
How could I help with data?
thanks

[Unai-HP]

Hi. As said, I've been trying to develop the RL model using some papers. However, I was too optimistic with what I could achieve. I've tried a lot of ways and unfortunately all the results are quite unusable. While doing so I broke the pc I was using for the training (burned some capacitors) so I had to stop the attempts. I'll be returning to this when I have more time, but I can't promise anything. Regarding the data, don't worry, it's not needed. The model trains itself with a simulation (creating a base model), and then it would specialize while being used in xDrip (or at least that was my original idea).

So, I'm sorry to say that the progress is none for now, unfortunately.