Plots doesn't find correct backend for PyPlots

[lewisl]

Details

There are too many conflicting ways to set the backend for Matplotlib. It is likely on MacOS that one ends up with multiple instances of Matplotlib. The only reliable way to set a backend seems to be to set the environment variable as
export MPLBACKEND=qt5agg
That's what I've done.

Now, let's see if that has "taken". You'll see below that when I run PyPlot directly, it reports the correct backend.

julia> using PyPlot

julia> rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams");

julia> rcParams["backend"]
"Qt5Agg"

Note that I get no report of a conflicting or unsupported backend and no complaints about TkAgg. I can plot successfully.

Now, let's see what happens in Plots.

julia> using Plots

julia> pyplot()
┌ Warning: PyPlot is using tkagg backend, which is known to cause crashes on MacOS (#410); use the MPLBACKEND environment variable to request a different backend.
└ @ PyPlot ~/.julia/packages/PyPlot/XHEG0/src/init.jl:192
Plots.PyPlotBackend()

julia> PyPlot.backend
"TkAgg"

Note that Plots seems to erroneously report the wrong backend. It is not clear where Plots is getting TkAgg. I have Qt5Agg selected in every place matplotlib can look.

Here it is in Python to verify that the version of Python used by my PyCall (and hence, pyplot within Plots).

Python 3.9.4 (default, Apr  9 2021, 09:32:38)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> import matplotlib.pyplot as plt
>>> import numpy as np
>>> plt.plot(np.arange(1,5))
[<matplotlib.lines.Line2D object at 0x7fd1f7d70190>]
>>> plt.show()  # sure enough simple plot shows up!
>>> plt.get_backend()
'Qt5Agg'
>>>

Here I verify the version of Python being used by my Julia installation:

julia> using PyCall

julia> PyCall.pyversion
v"3.9.4"

julia> PyCall.libpython
"/opt/miniconda3/lib/libpython3.9.dylib"

Let me know if there is anything else I can provide.

Backends

This bug occurs on ( insert x below )

Backend	yes	no	untested
gr (default)
pyplot	x
plotly
plotlyjs
pgfplotsx
inspectdr

Versions

Plots.jl version: v1.16.0
Backend version (]st -m <backend(s)>): PyPlot v2.9.0
Output of versioninfo():
Julia Version 1.6.2
Commit 1b93d53fc4 (2021-07-14 15:36 UTC)
Platform Info:
OS: macOS (x86_64-apple-darwin18.7.0)
CPU: Intel(R) Core(TM) i7-8557U CPU @ 1.70GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

[lewisl]

Here is an update. Looking around in source code for the pyplots backend I found that Plots imports (from somewhere) what it believes is the rcparams for Matplotlib.

So, I did this:

julia> Plots.pyrcparams["backend"]="Qt5Agg"
"Qt5Agg"

julia> pyplot()
Plots.PyPlotBackend()

julia> plot(1:5)

Note that I set a new value for the backend before setting PyPlot as the backend. Then, when I set the backend Plots doesn't complain. I can go ahead and plot and it works perfectly.

This is a deep mystery. Plots calls PyPlot to read the rcparams. But, what it gets back is NOT the same as what PyPlot itself reports. In the init.jl for PyPlot, it looks up the backend for matplotlib. Here is what it reports on my system:

julia> using PyPlot

julia> PyPlot.backend
"qt5agg"

julia> PyPlot.gui
:qt5

PyPlot.version
v"3.4.2".   # note: this is the version of matplotlib, not PyPlot

Finally, conda reports my version of Matplotlib which agrees with PyPlot: matplotlib 3.4.2 py39hecd8cb5_0

The above is sort of an OK work around, but really shouldn't be what I need to do. Perhaps the problem is in where PyPlot reads the rcparams--maybe it reads different versions?

[isentropic]

The RC params Plots uses is under Plots.pyrcparam I believe do is it also different?

[lewisl]

Yes, I've looked at this. It reports that it is using "TkAgg"

The problem remains: why does Plots get it wrong when PyPlot package in Julia and Python matplotlib get it right?

Where does Plots get this value in Plots.rcparams? I believe it has some defaults it might carry around in the package. I'll check.

How can it be reset?
This is easy. It's a dict. So I set the backend manually: Plots.pyrcparams["backend"] = "Qt5Agg"

So, I now can run the following:

using Plots
pyplot()   # results in TkAgg warning in the original post
Plots.pyrcparams["backend"] = "Qt5Agg"
# now to see if this has "taken" switch to gr and switch back to pyplot:
gr()
pyplot()    # aha--no warning occurs
plot(1:5)   # the plot window is drawn like a pyplot window and it works. I have to assume PyPlot and matplotlib are happy...

[lewisl]

I am very sorry. I must confirm that the bug IS in Plots.

PyPlot package is reading the correct backend. Here we see that:

julia> using PyPlot

julia> rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams")
PyCall.PyDict{PyCall.PyAny, PyCall.PyAny, true} with 310 entries:
  "backend"                => "Qt5Agg"
  "webagg.port"            => 8988
# Note: many more params...

Again, I repeat myself. Plots does NOT read the rcparams correctly or it seems to use some defaults from some unspecified place. This is surprising because in the code it is clear that Plots attempts to have PyPlot read the params. Perhaps Plots is calling the wrong function in PyPlot.

If I run using PyPlot and plot something, it plots correctly and shows the matplotlib backend as above. If I immediately run using Plots and pyplot(). There is no warning and plots are correctly rendered. So, somehow PyPlot is leaving a data structure accessible to Julia, which Plots then reads.

But, if I load Plots first then the pyplot backend always uses the wrong backend.

I will also note that Plots does NOT pickup the value of the MPLBACKEND variable.

Here you can see that in addition to choosing the backend in the .matplotlibrc file I have set the environment variable, which Julia shows in the ENV dict:

julia> ENV["MPLBACKEND"]
"qt5agg"

So, sorry Virginia--there is indeed a bug.

I am at the end of my limit to help debug Plots. If I can provide any more proof or diagnostic information I absolutely will.

[isentropic]

So after

using Plots
pyplot()   # results in TkAgg warning in the original post
Plots.pyrcparams["backend"] = "Qt5Agg"

Can you use pyplot backend as usual?

[lewisl]

Yes, if I manually set the rcparams backend as shown. The problem is that until some recent updates—of which there have been many—this was never necessary. Why not look through all of the commits of the past 8 weeks and find the problem? I think I have established that the problem is not in PyPlot package or in Matplotlib. From: Zhanibek ***@***.***> Reply-To: "JuliaPlots/Plots.jl" ***@***.***> Date: Wednesday, September 29, 2021 at 6:57 PM To: "JuliaPlots/Plots.jl" ***@***.***> Cc: Lewis Levin ***@***.***>, Author ***@***.***> Subject: Re: [JuliaPlots/Plots.jl] Plots doesn't find correct backend for PyPlots (#3802) So after using Plots pyplot() # results in TkAgg warning in the original post Plots.pyrcparams["backend"] = "Qt5Agg" Can you use pyplot backend as usual? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#3802 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIYWLL6MI5DGH2WKIIK4RLUEO7YBANCNFSM5D6RJAIQ>.

[isentropic]

I could help you out if you find the commit or tag after which this has stopped working. For now, I can only suggest to sticking to this workaround.
I do not think that backend initialization or anything, in particular, has changed recently. I do not have a mac device so my help is limited

[braydenware]

I am seeing similar behavior. On Mac, PyPlot detects qt5agg automatically, while Plots with the pyplot backend does not detect it. In my case, setting the environment variable MPLBACKEND=qt5agg made no difference.

I dug into the backend selection logic in Plots to try to find a minimal example of unexpected behavior. This is what I've found:

julia> import Plots
julia> import PyCall
julia> PyCall.pyexists("matplotlib.backends.backend_qt5agg")
false

while (in a fresh julia session of course) the behavior is different if the order is swapped:

julia> import PyCall
julia> PyCall.pyexists("matplotlib.backends.backend_qt5agg")
true
julia> import Plots
julia> PyCall.pyexists("matplotlib.backends.backend_qt5agg")
true

It is not enough to just import PyCall first.

julia> import PyCall
julia> import Plots
julia> PyCall.pyexists("matplotlib.backends.backend_qt5agg")
false

In my case, importing PyCall and calling PyCall.pyexists before loading Plots was sufficient; afterwards, switching to the pyplot backend of Plots and plotting works as expected.

(My environment is a M1 Mac with a fresh install of julia 1.6.3, with the latest tagged versions of PyCall, PyPlot and Plots as of today.)

[isentropic]

This is definitely weird, let me check it out

[isentropic]

As much as I would like to fix this, for now I cannot understand why such thing happens as Plots does not set any RC or launch options for PyPlot.

[isentropic]

All we do is import pyplot module expecting it would work.

[isentropic]

Maybe this is related too
#3416
#3800

[isentropic]

I do not have a mac system where I can reproduce this issue, but let's perhaps collectively gather some more facts and try to dig into this?
Maybe a starting point for all would be to get a new Julia installation with a clean environment. Also, let's just use python and pyplot that has been provided by the conda.jl (auto installation). Conda.jl is the most supported pyplot Plots supports.

[isentropic]

Here is an update. Looking around in source code for the pyplots backend I found that Plots imports (from somewhere) what it believes is the rcparams for Matplotlib.

So, I did this:

julia> Plots.pyrcparams["backend"]="Qt5Agg"
"Qt5Agg"

julia> pyplot()
Plots.PyPlotBackend()

julia> plot(1:5)

Note that I set a new value for the backend before setting PyPlot as the backend. Then, when I set the backend Plots doesn't complain. I can go ahead and plot and it works perfectly.

This is a deep mystery. Plots calls PyPlot to read the rcparams. But, what it gets back is NOT the same as what PyPlot itself reports. In the init.jl for PyPlot, it looks up the backend for matplotlib. Here is what it reports on my system:

julia> using PyPlot

julia> PyPlot.backend
"qt5agg"

julia> PyPlot.gui
:qt5

PyPlot.version
v"3.4.2".   # note: this is the version of matplotlib, not PyPlot

Finally, conda reports my version of Matplotlib which agrees with PyPlot: matplotlib 3.4.2 py39hecd8cb5_0

The above is sort of an OK work around, but really shouldn't be what I need to do. Perhaps the problem is in where PyPlot reads the rcparams--maybe it reads different versions?

Would you let me know where that is?

[lewisl]

If you just search the entire repo for rcparams you will find it. Only 1 or 2 hits in each (PyPlot, Plots) repo. From: Zhanibek ***@***.***> Reply-To: "JuliaPlots/Plots.jl" ***@***.***> Date: Monday, October 11, 2021 at 8:18 PM To: "JuliaPlots/Plots.jl" ***@***.***> Cc: Lewis Levin ***@***.***>, Author ***@***.***> Subject: Re: [JuliaPlots/Plots.jl] Plots doesn't find correct backend for PyPlots (#3802) Here is an update. Looking around in source code for the pyplots backend I found that Plots imports (from somewhere) what it believes is the rcparams for Matplotlib. So, I did this: julia> Plots.pyrcparams["backend"]="Qt5Agg" "Qt5Agg" julia> pyplot() Plots.PyPlotBackend() julia> plot(1:5) Note that I set a new value for the backend before setting PyPlot as the backend. Then, when I set the backend Plots doesn't complain. I can go ahead and plot and it works perfectly. This is a deep mystery. Plots calls PyPlot to read the rcparams. But, what it gets back is NOT the same as what PyPlot itself reports. In the init.jl for PyPlot, it looks up the backend for matplotlib. Here is what it reports on my system: julia> using PyPlot julia> PyPlot.backend "qt5agg" julia> PyPlot.gui :qt5 PyPlot.version v"3.4.2". # note: this is the version of matplotlib, not PyPlot Finally, conda reports my version of Matplotlib which agrees with PyPlot: matplotlib 3.4.2 py39hecd8cb5_0 The above is sort of an OK work around, but really shouldn't be what I need to do. Perhaps the problem is in where PyPlot reads the rcparams--maybe it reads different versions? Would you let me know where that is? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#3802 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIYWLMWEETJ3O5ZZGNGEA3UGOSHVANCNFSM5D6RJAIQ>.

[lewisl]

Except it works by itself, but not when called by Plots. Also, it worked prior to a recent update to Plots (not sure which one). Or perhaps a recent update to PyPlot is the culprit. That’s why I suspected PyPlot might read the rcparams in more than one place and Plots is accessing something wrong. But, I’d still suspect Plots is accessing the wrong rcparams or messing with it because PyPlot works using the same Python and the same matplotlib and the same user specific settings in the home directory. I’ll help since I “started it”. I sure hate the standard Anaconda super-pig installation. How to make Python + dependencies = 6G. No wonder Anaconda can make a living on support contracts. They don’t break any open source rules and they contribute/maintain code but they sure violate the spirit of open source (but, then so does everyone else…). Let’s just call it commercial with some voluntary license sharing and be done with whole false ideology of it. (OK, you didn’t need that commentary…). I use miniconda so I get the same distro of Python et al, but only 4G of it. So, at great pain and time I’ll delete my Python and my Julia (so many packages) and start over. I’ll let you know what happens. From: Zhanibek ***@***.***> Reply-To: "JuliaPlots/Plots.jl" ***@***.***> Date: Monday, October 11, 2021 at 8:13 PM To: "JuliaPlots/Plots.jl" ***@***.***> Cc: Lewis Levin ***@***.***>, Author ***@***.***> Subject: Re: [JuliaPlots/Plots.jl] Plots doesn't find correct backend for PyPlots (#3802) All we do is import pyplot module expecting it would work. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#3802 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIYWLN2EUKCSL4FPLOLRQTUGORWVANCNFSM5D6RJAIQ>.

[lewisl]

Embedded as >> Like I said, I’ll start clean and see how it goes. Assuming that fixes it (decent probability as these things go), we won’t know what caused it to begin with. Just strengthens the case for Julia to rely on fewer and smaller/simpler dependencies. It’s not like we really need more and more plotting APIs with their own involuted underlying stacks. How about one that’s really good? Time for the Julia powers-that-be to bite the bullet and do some of the lower level (hard) work to be a bit less of a ‘free-rider’ (sure, that’s the point of shared code and OSS, but it clearly has its downsides). Sometimes a whole generation of code needs a purging; benefit of new projects—but comes with huge obstacle of upfront cost and being a bit deficient for a while. Julia offers enough direct benefits that the community has weathered both disadvantages—now much reduced—with great success and benefits. From: Zhanibek ***@***.***> Reply-To: "JuliaPlots/Plots.jl" ***@***.***> Date: Monday, October 11, 2021 at 8:14 PM To: "JuliaPlots/Plots.jl" ***@***.***> Cc: Lewis Levin ***@***.***>, Author ***@***.***> Subject: Re: [JuliaPlots/Plots.jl] Plots doesn't find correct backend for PyPlots (#3802) Maybe this is related too #3416<#3416> >> well this sort of blamed Julia 1.6 but I’ve had everything working on 1.6.0, 1.6.1 and 1.6.2 #3800<#3800> >> well this one is about GR, which is a different and much skinnier stack — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#3802 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIYWLPH2XYQ2OXM2AH5S53UGOR2JANCNFSM5D6RJAIQ>.

[lewisl]

I have done as requested: deleted my Julia installation, remove the ~/.julia directory to start over.
I installed Julia 1.6.3. I installed PyCall, PyPlot, and Plots.
I found out I needed to upgrade my matplotlib from 3.3.4 to 3.4.3, which I did.

Now, start Julia and attempt to plot with Plots to the pyplot() backend:

julia> using Plots

julia> pyplot()
┌ Warning: No working GUI backend found for matplotlib
└ @ PyPlot ~/.julia/packages/PyPlot/XaELc/src/init.jl:165
Plots.PyPlotBackend()

Let's verify if this is correct about the GUI backend using PyPlot directly (after quitting and restarting Julia):

julia> using PyPlot

julia> plot(1:10)
1-element Vector{PyCall.PyObject}:
 PyObject <matplotlib.lines.Line2D object at 0x7fa57133edf0>

julia> using PyCall

julia> pygui()
:qt5

Note: when the PyObject was returned, it also opened up a qt window containing the very impressive analytical plot of integers from 0 to 9.

So we see that PyPlot returns no complaint about gui backend. Per PyPlot documentation we use PyCall to query or set the gui and it reports that :qt5 is the symbol for the active backend.

Sorry to report that a totally clean install (worth doing from time to time despite the small-ish inconvenience) of Plots reveals that Plots integration with PyPlot appears to be broken on MacOS in a meaningful way.

[isentropic]

This is unfortunate, but unless someone with a will to fix this (or add a hack) comes around, there is nothing I can do. I do not have a macos system and do not want to.

[isentropic]

I still never found a line in the Plots codebase that could be responsible for this. To me it looks like that things should just work

[lewisl]

Anything else I can do to diagnose? Having reinstalled all of Julia, I’ll reinstall all of Python—occasional purging is good for the system. Anything else I might do to fix? I’ll also poke around in the code. It’s clear that there are multiple places the settings for matplotlib are kept—some/most in Python and packages and fewer in Julia packages. On Oct 20, 2021, at 7:51 PM, Zhanibek ***@***.******@***.***>> wrote: This is unfortunate, but unless someone with a will to fix this (or add a hack) comes around, there is nothing I can do. I do not have a macos system and do not want to. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#3802 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIYWLLAUQRMAC75LCCR6LLUH5527ANCNFSM5D6RJAIQ>.

[lewisl]

I reinstalled my Python with some difficulty because I chose not to use Anaconda or miniconda--my choice so my problem. But instead of 3.5G of Python, I have 350 mb of Python with matplotlib, numpy and other dependencies. (Just need to reinstall all the Jupyter stuff).

In any case, there is no problem finding the backend for pyplot() when using Plots. So, that's good.

I think that there can be multiple locations for the matplotlibrc file and this caused the original problem--so can be solved with Conda (indeed most people don't see this). Matplotlib has its own search order for these. By having one and only one I think the problem goes away. It is interesting that Plots got a different one than PyPlot and matplotlib itself. But, there is a way out...

Over several years, I have tried a variety of Python builds so I think I had lingering detritus from other python installs. Now I have the system install of 2.7, which is never used ever by anything, and one homebrew installation of Python 3.9.

Right now the Python Software Foundation doesn't have builds of Python for new M chip Macs and MacOS Big Sur 11.6. Soon, Apple will release Monterrey, which is version 12.x.

This will throw a loop for lots of open source projects because these are big changes coming all at once...

Hopefully Julia gets the chip support for free by using the latest version of LLVM on Macs...

[isentropic]

I hear you regarding the pain of python installation. My mantra is to just rely on Conda.jl and whatever matplotlib and python it ships by default.

Just a general awareness for everyone in the future:

The problem of installation of pyplot backend is not a new one. For anyone facing this issue in the future, I suggest to just go with Conda.jl without getting in trouble of having your own python, pip, conda complications.

I firmly believe that 3.5GB ain't that much of an overhead for the benefits of stability it provides. If you choose not so, it could be hard to determine the root of the problem as was in this issue.

[lewisl]

Overall, you are right. Specifically, the problem I encountered was in a miniconda installation of Python, which I removed to start over. On Oct 25, 2021, at 9:13 PM, Zhanibek ***@***.******@***.***>> wrote: I hear you regarding the pain of python installation. My mantra is to just rely on Conda.jl and whatever matplotlib and python it ships by default. Just a general awareness for everyone in the future: The problem of installation of pyplot backend is not a new one. For anyone facing this issue in the future, I suggest to just go with Conda.jl without getting in trouble of having your own python, pip, conda complications. I firmly believe that 3.5GB ain't that much of an overhead for the benefits of stability it provides. If you choose not so, it could be hard to determine the root of the problem as was in this issue. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub<#3802 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAIYWLLZTTMTDOVICKH7B33UIYTGFANCNFSM5D6RJAIQ>.

[lewisl]

Still closed.

Pointer: one can get nearly 2.5G out of a miniconda installation by eliminating use of Intel MKL floating point library. If you are doing your ML work in Julia rather than Python this will do no harm.

Do follow the instructions that Anaconda provides. it is important to install versions of several packages that are compiled to use OpenBlas. Their instructions are good. However, they forget to mention that after following the 2 (simple) conda commands, you can remove the MKL packages. Like Julia, Anaconda Python prefers to keep unused packages present in case you wish to change your mind. Entirely reasonable, but you gain the savings by removing the MKL packages entirely. I got rid of 2.5G. Still crazy big, but OK.

[avonmoll]

I found this thread after experiencing the same behavior noted by @lewisl and @braydenware (I'm also on macos). So I tried @isentropic's suggestion and tried to rely only on Conda.jl. So, I tried removing PyCall and PyPlot, installed Conda.jl and ran Conda.update() a couple of times (also had issues with MKL python libs, which is separate), then set ENV["PYTHON"] = "" and added back in PyCall, called ] build PyCall, added PyPlot, and this is what I get:

julia> using Plots; pyplot()
Info: Installing pyqt package to avoid buggy tkagg backend.
Warning: No working GUI backend found for matplotlib
@ PyPlot ~/.julia/packages/PyPlot/XaELc/src/init.jl:165
Plots.PyPlotBackend()

Just to make sure, I tried to add pyqt to julia's conda environment: using Conda; Conda.add("pyqt"). It was already present.

Using PyPlot (without Plots.jl) works entirely as expected. And I get the exact behavior described by @braydenware, even after all of these steps.