Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Queries with and/or operators not working (on Elixir 1.6) #70

Open
sheharyarn opened this issue Jan 17, 2018 · 14 comments
Open

Queries with and/or operators not working (on Elixir 1.6) #70

sheharyarn opened this issue Jan 17, 2018 · 14 comments

Comments

@sheharyarn
Copy link
Contributor

sheharyarn commented Jan 17, 2018

On Elixir 1.6.0, where statements with and or or operators cause compilation errors. The where macro works fine when they are not used, but when they are, for some reason elixir expands the match fields to functions (which don't exist), throwing a compilation error.

Environment Details:

  • Elixir 1.6.0
  • Erlang 20.1
  • Amnesia 0.2.7 (Exquisite 0.1.7)
  • MacOS High Sierra 10.13.2

Reproducing the Issue:

A simple elixir application with amnesia set up, along with this DB/Table. This would not compile on Elixir 1.6.0, but if you remove the find/2 method, it will.

use Amnesia

defdatabase DB do
  deftable Post, [:user, :status, :content] do

    # Insert a Post
    def insert(user, status, content) do
      Amnesia.transaction do
        write(%DB.Post{user: user, status: status, content: content})
      end
    end


    # Find all posts by user
    def find(p_user) do
      Amnesia.transaction do
        where(user == p_user)
        |> Amnesia.Selection.values
      end
    end


    # Find all posts by user and status
    def find(p_user, p_status) do
      Amnesia.transaction do
        where(user == p_user and status == p_status)
        |> Amnesia.Selection.values
      end
    end

  end
end

Error:

Erlang/OTP 20 [erts-9.1] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]

Compiling 1 file (.ex)
warning: variable "user" does not exist and is being expanded to "user()", please use parentheses to remove the ambiguity or change the variable name
  lib/amnesia.ex:26

warning: variable "status" does not exist and is being expanded to "status()", please use parentheses to remove the ambiguity or change the variable name
  lib/amnesia.ex:26


== Compilation error in file lib/amnesia.ex ==
** (CompileError) lib/amnesia.ex:26: undefined function status/0
    (stdlib) lists.erl:1338: :lists.foreach/2
    lib/amnesia.ex:4: (module)
    (elixir) lib/kernel/parallel_compiler.ex:198: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/6

Everything else works as expected. This is a serious bug which also affects one of my projects; Que. Hoping this gets resolved soon!

@suprnova32
Copy link

I was just about to report this. In order to work around this, I had to modify my fork of Que and add some dirty hacks: https://github.com/AlloyCI/que/commit/593dc65c3405c0f5381a091ce2703c3401618485

AlloyCI really depends on this, so hoping this gets resolved soon, I don't want to rely on a dirty hack.

@meh
Copy link
Owner

meh commented Jan 19, 2018

This sounds like a bug in Elixir to me, looking into it.

@suprnova32
Copy link

@meh while investigating the errors, it seemed to me that they might be coming from a bug in expanding the and and or macros. I haven't checked if there are any differences between Elixir 1.5 and 1.6, but it would seem like a good place to start.

@meh
Copy link
Owner

meh commented Jan 21, 2018

@supernova32 yeah, my suspicion is something changed slightly in the AST between 1.5 and 1.6, I just need to find the time to look into that, unless someone else beats me to it, this sounds like it shouldn't have happened.

@josevalim
Copy link

The issue is that exquisite is relying on a macro to expand to a certain format which is a no-no since Elixir can change what macro expands to at any moment.

The way Ecto approaches this is by traversing the known nodes before expanding them. So for example, if you know how to handle the {:and, _, [x, y]} AST, you handle that first, and then you have a catch all clause that try to expand the AST once and try again. If the AST does not expand, then you raise. Here is this particular bit in Ecto. This way you are not coupled to how Elixir expands some macros (and you can even add your own nodes that does not exist in Elixir).

@meh
Copy link
Owner

meh commented Jan 24, 2018

@josevalim actually now that I look into it a little more, it seems to have nothing to do with the AST.

Using the following as example:

    from = {{2013,1,1},{1,1,1}}
    to   = {{2013,2,2},{1,1,1}}

    s = Exquisite.match { a, b },
      where:  a >= from and b <= to,
      select: 2

Elixir 1.5:

{{:a, [line: 27], nil}, {:b, [line: 27], nil}}
[where: {:and, [line: 28],
  [{:>=, [line: 28], [{:a, [line: 28], nil}, {:from, [line: 28], nil}]},
   {:<=, [line: 28], [{:b, [line: 28], nil}, {:to, [line: 28], nil}]}]},
 select: 2]

[{:{}, [],
  [{:"$1", :"$2"},
   [{:{}, [],
     [:andalso,
      {:{}, [],
       [:>=, :"$1",
        {{:., [], [{:__aliases__, [alias: false], [:Exquisite]}, :convert]}, []
         [{:from, [line: 28], nil}]}]},
      {:{}, [],
       [:"=<", :"$2",
        {{:., [], [{:__aliases__, [alias: false], [:Exquisite]}, :convert]}, []
         [{:to, [line: 28], nil}]}]}]}], [2]]}]

Elixir 1.6:

{{:a, [line: 27], nil}, {:b, [line: 27], nil}}
[
  where: {:and, [line: 28],
   [
     {:>=, [line: 28], [{:a, [line: 28], nil}, {:from, [line: 28], nil}]},
     {:<=, [line: 28], [{:b, [line: 28], nil}, {:to, [line: 28], nil}]}
   ]},
  select: 2
]

[
  {:{}, [],
   [
     {:"$1", :"$2"},
     [
       {{:., [], [{:__aliases__, [alias: false], [:Exquisite]}, :convert]}, [],
        [
          {:case, [optimize_boolean: true],
           [
             {:>=, [line: 28],
              [{:a, [line: 28], nil}, {:from, [line: 28], nil}]},
             [
               do: [
                 {:->, [],
                  [
                    [true],
                    {:<=, [line: 28],
                     [{:b, [line: 28], nil}, {:to, [line: 28], nil}]}
                  ]},
                 {:->, [], [[false], false]},
                 {:->, [],
                  [
                    [{:other, [counter: -576460752303423484], Kernel}],
                    {{:., [], [:erlang, :error]}, [],
                     [
                       {{:., [],
                         [
                           {:__aliases__,
                            [alias: false, counter: -576460752303423484],
                            [:BadBooleanError]},
                           :exception
                         ]}, [],
                        [
                          [
                            operator: :and,
                            term: {:other, [counter: -576460752303423484],
                             Kernel}
                          ]
                        ]}
                     ]}
                  ]}
               ]
             ]
           ]}
        ]}
     ],
     [2]
   ]}
]

I don't see any difference in the input (unless I'm blind) so it must be some change in macro expansion, did anything in Macro.escape change?

@ourway
Copy link

ourway commented Mar 26, 2018

Any Updates on this?

@sheharyarn
Copy link
Contributor Author

Bump. Would really like to see this issue resolved soon. @meh @josevalim Is there anything I can do to get this fixed? What do you think the problem is here?

@astutecat
Copy link

astutecat commented May 29, 2018

I also have this issue occurring for me, sadly - same symptoms on elixir 1.6

e: also happy to help where I can

@jonathanleang
Copy link

FIY,
change

  •        MRoom.where(lobby_name == "rooms" and topic == key)
    

to

  •       MRoom.match(lobby_name: "rooms", topic: key)
    

works for me

@DerTim1
Copy link

DerTim1 commented Aug 28, 2018

@jonathanleang Do you have a work around for where-queries with or and != in it? Or combinations of or and and?

@dmilith
Copy link

dmilith commented Sep 6, 2018

Exactly. Before I had:

hist = Amnesia.Selection.values History.where timestamp != -1

now.. I can't do it. I mean literally I can't get Amnesia to work on Elixir 1.7. It was just fine on 1.6. Amount of failures I got caused by CALLER or other stuff like this is staggering (exquisite as an example). I have to stay on Elixir 1.6 cause this is madness that I'm literally unable to fix my project now.

With amount of incompatibilities Elixir 1.7 is causing.. I don't understand why it's not Elixir 2.0 instead - since there are plenty of MAJOR changes in language.

@noizu
Copy link

noizu commented Nov 20, 2018

I'm able to run Amnesia with and/or queries on Elixir 1.7.4, Erlang 21 using my patched version of exquisite.

Patched Exquisite to handle and/or queries.
Diff of my exquisite branch and upstream

In Deps:
'''
{:exquisite, git: "https://github.com/noizu/exquisite.git", ref: "7a4a03d", override: true},
'''

@sheharyarn
Copy link
Contributor Author

I actually just ended up writing my own wrapper around mnesia, keeping it much closer to the original api.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

10 participants