From a4956eee407d2d744775d95afbc70d42f369db8f Mon Sep 17 00:00:00 2001
From: Zach Allaun <zach.allaun@gmail.com>
Date: Thu, 14 Sep 2023 09:46:13 -0400
Subject: [PATCH 01/20] Add `<|>` to the list of parsable but unused operators
 (#12932)

---
 lib/elixir/pages/references/operators.md | 45 ++++++++++++------------
 1 file changed, 23 insertions(+), 22 deletions(-)

diff --git a/lib/elixir/pages/references/operators.md b/lib/elixir/pages/references/operators.md
index 41642f0a9f5..608abe16d44 100644
--- a/lib/elixir/pages/references/operators.md
+++ b/lib/elixir/pages/references/operators.md
@@ -6,28 +6,28 @@ This document is a complete reference of operators in Elixir, how they are parse
 
 The following is a list of all operators that Elixir is capable of parsing, ordered from higher to lower precedence, alongside their associativity:
 
-Operator                                       | Associativity
----------------------------------------------- | -------------
-`@`                                            | Unary
-`.`                                            | Left
-`+` `-` `!` `^` `not`                          | Unary
-`**`                                           | Left
-`*` `/`                                        | Left
-`+` `-`                                        | Left
-`++` `--` `+++` `---` `..` `<>`                | Right
-`in` `not in`                                  | Left
-`\|>` `<<<` `>>>` `<<~` `~>>` `<~` `~>` `<~>`  | Left
-`<` `>` `<=` `>=`                              | Left
-`==` `!=` `=~` `===` `!==`                     | Left
-`&&` `&&&` `and`                               | Left
-`\|\|` `\|\|\|` `or`                           | Left
-`=`                                            | Right
-`&`                                            | Unary
-`=>` (valid only inside `%{}`)                 | Right
-`\|`                                           | Right
-`::`                                           | Right
-`when`                                         | Right
-`<-` `\\`                                      | Left
+Operator                                             | Associativity
+---------------------------------------------------- | -------------
+`@`                                                  | Unary
+`.`                                                  | Left
+`+` `-` `!` `^` `not`                                | Unary
+`**`                                                 | Left
+`*` `/`                                              | Left
+`+` `-`                                              | Left
+`++` `--` `+++` `---` `..` `<>`                      | Right
+`in` `not in`                                        | Left
+`\|>` `<\|>` `<<<` `>>>` `<<~` `~>>` `<~` `~>` `<~>` | Left
+`<` `>` `<=` `>=`                                    | Left
+`==` `!=` `=~` `===` `!==`                           | Left
+`&&` `&&&` `and`                                     | Left
+`\|\|` `\|\|\|` `or`                                 | Left
+`=`                                                  | Right
+`&`                                                  | Unary
+`=>` (valid only inside `%{}`)                       | Right
+`\|`                                                 | Right
+`::`                                                 | Right
+`when`                                               | Right
+`<-` `\\`                                            | Left
 
 ## General operators
 
@@ -129,6 +129,7 @@ The following is a table of all the operators that Elixir is capable of parsing,
   * `<~`
   * `~>`
   * `<~>`
+  * `<|>`
   * `+++`
   * `---`
 

From d87aadf8bd280d4ac969a6825637fcbd1e412f81 Mon Sep 17 00:00:00 2001
From: Zach Allaun <zach.allaun@gmail.com>
Date: Fri, 15 Sep 2023 09:05:49 -0400
Subject: [PATCH 02/20] Fix spec in `Code.Typespec.fetch_types/1` and
 `fetch_callbacks/1` (#12933)

The guards for these functions were already allowing binaries, which
allows either a file name or a binary containing the BEAM object code to
be passed in. The specs, however, only specified `module`. (Note that
`fetch_types/1` already specified `module | binary`.)
---
 lib/elixir/lib/code/typespec.ex | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/elixir/lib/code/typespec.ex b/lib/elixir/lib/code/typespec.ex
index d465403cfcf..7cda6db78b0 100644
--- a/lib/elixir/lib/code/typespec.ex
+++ b/lib/elixir/lib/code/typespec.ex
@@ -121,7 +121,7 @@ defmodule Code.Typespec do
   located by the runtime system. The types will be in the Erlang
   Abstract Format.
   """
-  @spec fetch_specs(module) :: {:ok, [tuple]} | :error
+  @spec fetch_specs(module | binary) :: {:ok, [tuple]} | :error
   def fetch_specs(module) when is_atom(module) or is_binary(module) do
     case typespecs_abstract_code(module) do
       {:ok, abstract_code} ->
@@ -142,7 +142,7 @@ defmodule Code.Typespec do
   which can be located by the runtime system. The types will be
   in the Erlang Abstract Format.
   """
-  @spec fetch_callbacks(module) :: {:ok, [tuple]} | :error
+  @spec fetch_callbacks(module | binary) :: {:ok, [tuple]} | :error
   def fetch_callbacks(module) when is_atom(module) or is_binary(module) do
     case typespecs_abstract_code(module) do
       {:ok, abstract_code} ->

From 9a3d1a03212a8b953a450cc4ec3c5f1444362715 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jos=C3=A9=20Valim?= <jose.valim@dashbit.co>
Date: Fri, 15 Sep 2023 16:31:41 +0200
Subject: [PATCH 03/20] Include open_delimiter in TokenMissingError

---
 lib/elixir/lib/exception.ex                   |  9 +++++++-
 lib/elixir/src/elixir_tokenizer.erl           |  3 ++-
 lib/elixir/test/elixir/kernel/errors_test.exs | 21 ++++++++++++-------
 3 files changed, 23 insertions(+), 10 deletions(-)

diff --git a/lib/elixir/lib/exception.ex b/lib/elixir/lib/exception.ex
index 75bede365ef..53aaa4e7f26 100644
--- a/lib/elixir/lib/exception.ex
+++ b/lib/elixir/lib/exception.ex
@@ -1040,7 +1040,14 @@ defmodule TokenMissingError do
 
   """
 
-  defexception [:file, :line, :snippet, :column, description: "expression is incomplete"]
+  defexception [
+    :file,
+    :line,
+    :snippet,
+    :column,
+    :open_delimiter,
+    description: "expression is incomplete"
+  ]
 
   @impl true
   def message(%{
diff --git a/lib/elixir/src/elixir_tokenizer.erl b/lib/elixir/src/elixir_tokenizer.erl
index ebd4b403015..851807d658b 100644
--- a/lib/elixir/src/elixir_tokenizer.erl
+++ b/lib/elixir/src/elixir_tokenizer.erl
@@ -149,7 +149,8 @@ tokenize([], EndLine, Column, #elixir_tokenizer{terminators=[{Start, {StartLine,
   Hint = missing_terminator_hint(Start, End, Scope),
   Message = "missing terminator: ~ts (for \"~ts\" starting at line ~B)",
   Formatted = io_lib:format(Message, [End, Start, StartLine]),
-  error({?LOC(EndLine, Column), [Formatted, Hint], []}, [], Scope, Tokens);
+  Meta = [{open_delimiter, Start} | ?LOC(EndLine, Column)],
+  error({Meta, [Formatted, Hint], []}, [], Scope, Tokens);
 
 tokenize([], Line, Column, #elixir_tokenizer{} = Scope, Tokens) ->
   #elixir_tokenizer{ascii_identifiers_only=Ascii, warnings=Warnings} = Scope,
diff --git a/lib/elixir/test/elixir/kernel/errors_test.exs b/lib/elixir/test/elixir/kernel/errors_test.exs
index 4993a4e27fc..1dea9602fec 100644
--- a/lib/elixir/test/elixir/kernel/errors_test.exs
+++ b/lib/elixir/test/elixir/kernel/errors_test.exs
@@ -469,12 +469,15 @@ defmodule Kernel.ErrorsTest do
   end
 
   test "invalid fn args" do
-    assert_eval_raise TokenMissingError,
-                      [
-                        "nofile:1:5:",
-                        ~r/missing terminator: end \(for "fn" starting at line 1\)/
-                      ],
-                      ~c"fn 1"
+    exception =
+      assert_eval_raise TokenMissingError,
+                        [
+                          "nofile:1:5:",
+                          ~r/missing terminator: end \(for "fn" starting at line 1\)/
+                        ],
+                        ~c"fn 1"
+
+    assert exception.open_delimiter == :fn
   end
 
   test "invalid escape" do
@@ -997,16 +1000,18 @@ defmodule Kernel.ErrorsTest do
   ## Helpers
 
   defp assert_eval_raise(given_exception, messages, source) do
-    e =
+    exception =
       assert_raise given_exception, fn ->
         Code.eval_string(source)
       end
 
-    error_msg = Exception.format(:error, e, [])
+    error_msg = Exception.format(:error, exception, [])
 
     for msg <- messages do
       assert error_msg =~ msg
     end
+
+    exception
   end
 
   defp assert_compile_error(messages, string) do

From 646577af50934e38b05dd847ebbf7f47b98e30ea Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jos=C3=A9=20Valim?= <jose.valim@dashbit.co>
Date: Fri, 15 Sep 2023 18:14:51 +0200
Subject: [PATCH 04/20] Add opening_delimiter to token missing error

---
 lib/elixir/lib/exception.ex                   | 2 +-
 lib/elixir/src/elixir_tokenizer.erl           | 2 +-
 lib/elixir/test/elixir/kernel/errors_test.exs | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/elixir/lib/exception.ex b/lib/elixir/lib/exception.ex
index 53aaa4e7f26..ccfe2366790 100644
--- a/lib/elixir/lib/exception.ex
+++ b/lib/elixir/lib/exception.ex
@@ -1045,7 +1045,7 @@ defmodule TokenMissingError do
     :line,
     :snippet,
     :column,
-    :open_delimiter,
+    :opening_delimiter,
     description: "expression is incomplete"
   ]
 
diff --git a/lib/elixir/src/elixir_tokenizer.erl b/lib/elixir/src/elixir_tokenizer.erl
index 851807d658b..d2958f5afd4 100644
--- a/lib/elixir/src/elixir_tokenizer.erl
+++ b/lib/elixir/src/elixir_tokenizer.erl
@@ -149,7 +149,7 @@ tokenize([], EndLine, Column, #elixir_tokenizer{terminators=[{Start, {StartLine,
   Hint = missing_terminator_hint(Start, End, Scope),
   Message = "missing terminator: ~ts (for \"~ts\" starting at line ~B)",
   Formatted = io_lib:format(Message, [End, Start, StartLine]),
-  Meta = [{open_delimiter, Start} | ?LOC(EndLine, Column)],
+  Meta = [{opening_delimiter, Start} | ?LOC(EndLine, Column)],
   error({Meta, [Formatted, Hint], []}, [], Scope, Tokens);
 
 tokenize([], Line, Column, #elixir_tokenizer{} = Scope, Tokens) ->
diff --git a/lib/elixir/test/elixir/kernel/errors_test.exs b/lib/elixir/test/elixir/kernel/errors_test.exs
index 1dea9602fec..46f8cdc4115 100644
--- a/lib/elixir/test/elixir/kernel/errors_test.exs
+++ b/lib/elixir/test/elixir/kernel/errors_test.exs
@@ -477,7 +477,7 @@ defmodule Kernel.ErrorsTest do
                         ],
                         ~c"fn 1"
 
-    assert exception.open_delimiter == :fn
+    assert exception.opening_delimiter == :fn
   end
 
   test "invalid escape" do

From 3111a5b78a1be53bb8b9d3e05a24b082cd5a653e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Vin=C3=ADcius=20M=C3=BCller?= <vinigm.nho@gmail.com>
Date: Fri, 15 Sep 2023 16:31:48 +0000
Subject: [PATCH 05/20] Improve mismatched delimiter message (#12928)

---
 lib/elixir/lib/exception.ex                   | 211 +++++++++++++++++-
 lib/elixir/src/elixir_errors.erl              |  20 +-
 lib/elixir/src/elixir_tokenizer.erl           |  22 +-
 .../test/elixir/kernel/diagnostics_test.exs   | 201 +++++++++++++++++
 lib/elixir/test/elixir/kernel/parser_test.exs |  18 +-
 lib/iex/test/iex/interaction_test.exs         |   3 +-
 6 files changed, 435 insertions(+), 40 deletions(-)

diff --git a/lib/elixir/lib/exception.ex b/lib/elixir/lib/exception.ex
index ccfe2366790..39047275d7e 100644
--- a/lib/elixir/lib/exception.ex
+++ b/lib/elixir/lib/exception.ex
@@ -937,40 +937,227 @@ defmodule MismatchedDelimiterError do
   - `fn a -> )`
   """
 
+  @max_lines_shown 5
+
   defexception [
     :file,
     :line,
     :column,
     :end_line,
     :end_column,
+    :opening_delimiter,
+    :closing_delimiter,
     :snippet,
     description: "mismatched delimiter error"
   ]
 
   @impl true
   def message(%{
-        line: _start_line,
-        column: _start_column,
+        line: start_line,
+        column: start_column,
         end_line: end_line,
         end_column: end_column,
         description: description,
+        opening_delimiter: opening_delimiter,
+        closing_delimiter: _closing_delimiter,
         file: file,
         snippet: snippet
       }) do
-    snippet =
-      :elixir_errors.format_snippet(
-        {end_line, end_column},
-        file,
-        description,
-        snippet,
-        :error,
-        [],
-        nil
-      )
+    start_pos = {start_line, start_column}
+    end_pos = {end_line, end_column}
+    lines = String.split(snippet, "\n")
+    expected_delimiter = :elixir_tokenizer.terminator(opening_delimiter)
 
+    snippet = format_snippet(start_pos, end_pos, description, file, lines, expected_delimiter)
     format_message(file, end_line, end_column, snippet)
   end
 
+  defp format_snippet(
+         {start_line, _start_column} = start_pos,
+         {end_line, end_column} = end_pos,
+         description,
+         file,
+         lines,
+         expected_delimiter
+       )
+       when start_line < end_line do
+    max_digits = digits(end_line)
+    general_padding = max(2, max_digits) + 1
+    padding = n_spaces(general_padding)
+
+    relevant_lines =
+      if end_line - start_line < @max_lines_shown do
+        line_range(lines, start_pos, end_pos, padding, max_digits, expected_delimiter)
+      else
+        trimmed_inbetween_lines(
+          lines,
+          start_pos,
+          end_pos,
+          padding,
+          max_digits,
+          expected_delimiter
+        )
+      end
+
+    """
+     #{padding}#{red("error:")} #{pad_message(description, padding)}
+     #{padding}│
+    #{relevant_lines}
+     #{padding}│
+     #{padding}└─ #{Path.relative_to_cwd(file)}:#{end_line}:#{end_column}\
+    """
+  end
+
+  defp format_snippet(
+         {start_line, start_column},
+         {end_line, end_column},
+         description,
+         file,
+         lines,
+         expected_delimiter
+       )
+       when start_line == end_line do
+    max_digits = digits(end_line)
+    general_padding = max(2, max_digits) + 1
+    padding = n_spaces(general_padding)
+
+    line = Enum.fetch!(lines, end_line - 1)
+    formatted_line = [line_padding(end_line, max_digits), to_string(end_line), " │ ", line]
+
+    mismatched_closing_line =
+      [
+        n_spaces(start_column - 1),
+        red("│"),
+        mismatched_closing_delimiter(end_column - start_column, expected_delimiter)
+      ]
+
+    unclosed_delimiter_line =
+      [padding, " │ ", unclosed_delimiter(start_column)]
+
+    below_line = [padding, " │ ", mismatched_closing_line, "\n", unclosed_delimiter_line]
+
+    """
+     #{padding}#{red("error:")} #{pad_message(description, padding)}
+     #{padding}│
+    #{formatted_line}
+    #{below_line}
+     #{padding}│
+     #{padding}└─ #{Path.relative_to_cwd(file)}:#{end_line}:#{end_column}\
+    """
+  end
+
+  defp line_padding(line_number, max_digits) do
+    line_digits = digits(line_number)
+
+    spacing =
+      if line_digits == 1 do
+        max(2, max_digits)
+      else
+        max_digits - line_digits + 1
+      end
+
+    n_spaces(spacing)
+  end
+
+  defp n_spaces(n), do: String.duplicate(" ", n)
+
+  defp digits(number, acc \\ 1)
+  defp digits(number, acc) when number < 10, do: acc
+  defp digits(number, acc), do: digits(div(number, 10), acc + 1)
+
+  defp trimmed_inbetween_lines(
+         lines,
+         {start_line, start_column},
+         {end_line, end_column},
+         padding,
+         max_digits,
+         expected_delimiter
+       ) do
+    start_padding = line_padding(start_line, max_digits)
+    end_padding = line_padding(end_line, max_digits)
+    first_line = Enum.fetch!(lines, start_line - 1)
+    last_line = Enum.fetch!(lines, end_line - 1)
+
+    """
+    #{start_padding}#{start_line} │ #{first_line}
+     #{padding}│ #{unclosed_delimiter(start_column)}
+     ...
+    #{end_padding}#{end_line} │ #{last_line}
+     #{padding}│ #{mismatched_closing_delimiter(end_column, expected_delimiter)}\
+    """
+  end
+
+  defp line_range(
+         lines,
+         {start_line, start_column},
+         {end_line, end_column},
+         padding,
+         max_digits,
+         expected_delimiter
+       ) do
+    start_line = start_line - 1
+    end_line = end_line - 1
+
+    lines
+    |> Enum.slice(start_line..end_line)
+    |> Enum.zip_with(start_line..end_line, fn line, line_number ->
+      line_number = line_number + 1
+      start_line = start_line + 1
+      end_line = end_line + 1
+
+      line_padding = line_padding(line_number, max_digits)
+
+      cond do
+        line_number == start_line ->
+          [
+            line_padding,
+            to_string(line_number),
+            " │ ",
+            line,
+            "\n",
+            padding,
+            " │ ",
+            unclosed_delimiter(start_column)
+          ]
+
+        line_number == end_line ->
+          [
+            line_padding,
+            to_string(line_number),
+            " │ ",
+            line,
+            "\n",
+            padding,
+            " │ ",
+            mismatched_closing_delimiter(end_column, expected_delimiter)
+          ]
+
+        true ->
+          [line_padding, to_string(line_number), " │ ", line]
+      end
+    end)
+    |> Enum.intersperse("\n")
+  end
+
+  defp mismatched_closing_delimiter(end_column, expected_closing_delimiter),
+    do: [
+      n_spaces(end_column - 1),
+      red(~s/└ mismatched closing delimiter (expected "#{expected_closing_delimiter}")/)
+    ]
+
+  defp unclosed_delimiter(start_column),
+    do: [n_spaces(start_column - 1), red("└ unclosed delimiter")]
+
+  defp pad_message(message, padding), do: String.replace(message, "\n", "\n #{padding}")
+
+  defp red(string) do
+    if IO.ANSI.enabled?() do
+      [IO.ANSI.red(), string, IO.ANSI.reset()]
+    else
+      string
+    end
+  end
+
   defp format_message(file, line, column, message) do
     location = Exception.format_file_line_column(Path.relative_to_cwd(file), line, column)
     "mismatched delimiter found on " <> location <> "\n" <> message
diff --git a/lib/elixir/src/elixir_errors.erl b/lib/elixir/src/elixir_errors.erl
index 1e40f6e78fc..59135b1503d 100644
--- a/lib/elixir/src/elixir_errors.erl
+++ b/lib/elixir/src/elixir_errors.erl
@@ -373,11 +373,8 @@ parse_error(Location, File, Error, <<"[", _/binary>> = Full, Input) when is_bina
 
 %% Given a string prefix and suffix to insert the token inside the error message rather than append it
 parse_error(Location, File, {ErrorPrefix, ErrorSuffix}, Token, Input) when is_binary(ErrorPrefix), is_binary(ErrorSuffix), is_binary(Token) ->
-  Message = <<ErrorPrefix/binary, Token/binary, ErrorSuffix/binary >>,
-  case lists:keytake(error_type, 1, Location) of
-    {value, {error_type, mismatched_delimiter}, Loc} -> raise_mismatched_delimiter(Loc, File, Input, Message);
-    _ -> raise_snippet(Location, File, Input, 'Elixir.SyntaxError', Message)
-  end;
+  Message = <<ErrorPrefix/binary, Token/binary, ErrorSuffix/binary>>,
+  raise_snippet(Location, File, Input, 'Elixir.SyntaxError', Message);
 
 %% Misplaced char tokens (for example, {char, _, 97}) are translated by Erlang into
 %% the char literal (i.e., the token in the previous example becomes $a),
@@ -391,7 +388,10 @@ parse_error(Location, File, <<"syntax error before: ">>, <<$$, Char/binary>>, In
 %% Everything else is fine as is
 parse_error(Location, File, Error, Token, Input) when is_binary(Error), is_binary(Token) ->
   Message = <<Error/binary, Token/binary>>,
-  raise_snippet(Location, File, Input, 'Elixir.SyntaxError', Message).
+  case lists:keytake(error_type, 1, Location) of
+    {value, {error_type, mismatched_delimiter}, Loc} -> raise_mismatched_delimiter(Loc, File, Input, Message);
+    _ -> raise_snippet(Location, File, Input, 'Elixir.SyntaxError', Message)
+  end.
 
 parse_erl_term(Term) ->
   {ok, Tokens, _} = erl_scan:string(binary_to_list(Term)),
@@ -399,11 +399,9 @@ parse_erl_term(Term) ->
   Parsed.
 
 raise_mismatched_delimiter(Location, File, Input, Message) ->
-  {end_line, EndLine} = lists:keyfind(end_line, 1, Location),
-  {end_column, EndCol} = lists:keyfind(end_column, 1, Location),
-  {InputString, InputStartLine, _} = Input,
-  Snippet = snippet_line(InputString, [{line, EndLine}, {column, EndCol}], InputStartLine),
-  raise('Elixir.MismatchedDelimiterError', Message,  [{file, File}, {snippet, Snippet} | Location]).
+  {InputString, _, _} = Input,
+  InputBinary = iolist_to_binary(InputString),
+  raise('Elixir.MismatchedDelimiterError', Message,  [{file, File}, {snippet, InputBinary} | Location]).
 
 raise_reserved(Location, File, Input, Keyword) ->
   raise_snippet(Location, File, Input, 'Elixir.SyntaxError',
diff --git a/lib/elixir/src/elixir_tokenizer.erl b/lib/elixir/src/elixir_tokenizer.erl
index d2958f5afd4..85070770123 100644
--- a/lib/elixir/src/elixir_tokenizer.erl
+++ b/lib/elixir/src/elixir_tokenizer.erl
@@ -1,7 +1,7 @@
 -module(elixir_tokenizer).
 -include("elixir.hrl").
 -include("elixir_tokenizer.hrl").
--export([tokenize/1, tokenize/3, tokenize/4, invalid_do_error/1]).
+-export([tokenize/1, tokenize/3, tokenize/4, invalid_do_error/1, terminator/1]).
 
 -define(at_op(T),
   T =:= $@).
@@ -1417,15 +1417,17 @@ check_terminator({End, {EndLine, EndColumn, _}}, [{Start, {StartLine, StartColum
     End ->
       {ok, Scope#elixir_tokenizer{terminators=Terminators}};
 
-    ExpectedEnd ->
-      Suffix =
-        io_lib:format(
-          "\n\n    HINT: the \"~ts\" on line ~B is missing terminator \"~ts\"",
-          [Start, StartLine, ExpectedEnd]
-        ),
-      StartLoc = ?LOC(StartLine, StartColumn),
-      EndLoc = [{end_line, EndLine}, {end_column, EndColumn}, {error_type, mismatched_delimiter}],
-      {error, {StartLoc ++ EndLoc, {unexpected_token_or_reserved(End), Suffix}, [atom_to_list(End)]}}
+    _ExpectedEnd ->
+      Meta = [
+        {line, StartLine},
+        {column, StartColumn},
+        {end_line, EndLine},
+        {end_column, EndColumn},
+        {error_type, mismatched_delimiter},
+        {opening_delimiter, Start},
+        {closing_delimiter, End}
+     ],
+     {error, {Meta, unexpected_token_or_reserved(End), [atom_to_list(End)]}}
   end;
 
 check_terminator({'end', {Line, Column, _}}, [], #elixir_tokenizer{mismatch_hints=Hints}) ->
diff --git a/lib/elixir/test/elixir/kernel/diagnostics_test.exs b/lib/elixir/test/elixir/kernel/diagnostics_test.exs
index 9eb2f769c54..4dc230f2de2 100644
--- a/lib/elixir/test/elixir/kernel/diagnostics_test.exs
+++ b/lib/elixir/test/elixir/kernel/diagnostics_test.exs
@@ -10,6 +10,207 @@ defmodule Kernel.DiagnosticsTest do
     on_exit(fn -> Application.put_env(:elixir, :ansi_enabled, true) end)
   end
 
+  describe "mismatched delimiter" do
+    test "same line" do
+      output =
+        capture_raise(
+          """
+          [1, 2, 3, 4, 5, 6)
+          """,
+          MismatchedDelimiterError
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:1:18:
+                 error: unexpected token: )
+                 │
+               1 │ [1, 2, 3, 4, 5, 6)
+                 │ │                └ mismatched closing delimiter (expected "]")
+                 │ └ unclosed delimiter
+                 │
+                 └─ nofile:1:18\
+             """
+    end
+
+    test "two-line span" do
+      output =
+        capture_raise(
+          """
+          [a, b, c
+           d, f, g}
+          """,
+          MismatchedDelimiterError
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:2:9:
+                 error: unexpected token: }
+                 │
+               1 │ [a, b, c
+                 │ └ unclosed delimiter
+               2 │  d, f, g}
+                 │         └ mismatched closing delimiter (expected "]")
+                 │
+                 └─ nofile:2:9\
+             """
+    end
+
+    test "many-line span" do
+      output =
+        capture_raise(
+          """
+              [ a,
+            b,
+            c,
+            d
+            e )
+          """,
+          MismatchedDelimiterError
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:5:5:
+                 error: unexpected token: )
+                 │
+               1 │     [ a,
+                 │     └ unclosed delimiter
+               2 │   b,
+               3 │   c,
+               4 │   d
+               5 │   e )
+                 │     └ mismatched closing delimiter (expected "]")
+                 │
+                 └─ nofile:5:5\
+             """
+
+      output =
+        capture_raise(
+          """
+          fn always_forget_end ->
+            IO.inspect(2 + 2) + 2
+          )
+          """,
+          MismatchedDelimiterError
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:3:1:
+                 error: unexpected token: )
+                 │
+               1 │ fn always_forget_end ->
+                 │ └ unclosed delimiter
+               2 │   IO.inspect(2 + 2) + 2
+               3 │ )
+                 │ └ mismatched closing delimiter (expected "end")
+                 │
+                 └─ nofile:3:1\
+             """
+    end
+
+    test "trim inbetween lines if too many" do
+      output =
+        capture_raise(
+          """
+          [ :a,
+            :b,
+            :c,
+            :d,
+            :e,
+            :f,
+            :g,
+            :h
+          )
+          """,
+          MismatchedDelimiterError
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:9:1:
+                 error: unexpected token: )
+                 │
+               1 │ [ :a,
+                 │ └ unclosed delimiter
+              ...
+               9 │ )
+                 │ └ mismatched closing delimiter (expected "]")
+                 │
+                 └─ nofile:9:1\
+             """
+    end
+
+    test "pads according to line number digits" do
+      output =
+        capture_raise(
+          """
+          [ a,
+          #{String.duplicate("\n", 10)}
+            b )
+          """,
+          MismatchedDelimiterError
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:13:5:
+                 error: unexpected token: )
+                 │
+               1 │ [ a,
+                 │ └ unclosed delimiter
+              ...
+              13 │   b )
+                 │     └ mismatched closing delimiter (expected "]")
+                 │
+                 └─ nofile:13:5\
+             """
+
+      output =
+        capture_raise(
+          """
+          [ a,
+          #{String.duplicate("\n", 400)}
+            b )
+          """,
+          MismatchedDelimiterError
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:403:5:
+                  error: unexpected token: )
+                  │
+                1 │ [ a,
+                  │ └ unclosed delimiter
+              ...
+              403 │   b )
+                  │     └ mismatched closing delimiter (expected "]")
+                  │
+                  └─ nofile:403:5\
+             """
+
+      output =
+        capture_raise(
+          """
+          #{String.duplicate("\n", 97)}
+          [ a,
+          #{String.duplicate("\n", 6)}
+            b )
+          """,
+          MismatchedDelimiterError
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:107:5:
+                  error: unexpected token: )
+                  │
+               99 │ [ a,
+                  │ └ unclosed delimiter
+              ...
+              107 │   b )
+                  │     └ mismatched closing delimiter (expected "]")
+                  │
+                  └─ nofile:107:5\
+             """
+    end
+  end
+
   describe "compile-time exceptions" do
     test "SyntaxError (snippet)" do
       expected = """
diff --git a/lib/elixir/test/elixir/kernel/parser_test.exs b/lib/elixir/test/elixir/kernel/parser_test.exs
index d84cfeace8b..bc9091fbeed 100644
--- a/lib/elixir/test/elixir/kernel/parser_test.exs
+++ b/lib/elixir/test/elixir/kernel/parser_test.exs
@@ -946,7 +946,8 @@ defmodule Kernel.ParserTest do
         [
           "nofile:1:9:",
           "unexpected token:",
-          "HINT: the \"fn\" on line 1 is missing terminator \"end\""
+          "└ unclosed delimiter",
+          "└ mismatched closing delimiter"
         ],
         ~c"fn a -> )"
       )
@@ -955,7 +956,8 @@ defmodule Kernel.ParserTest do
         [
           "nofile:1:16:",
           "unexpected token:",
-          "HINT: the \"do\" on line 1 is missing terminator \"end\""
+          "└ unclosed delimiter",
+          "└ mismatched closing delimiter"
         ],
         ~c"defmodule A do ]"
       )
@@ -964,7 +966,8 @@ defmodule Kernel.ParserTest do
         [
           "nofile:1:9:",
           "unexpected token:",
-          "HINT: the \"(\" on line 1 is missing terminator \")\""
+          "└ unclosed delimiter",
+          "└ mismatched closing delimiter"
         ],
         ~c"(1, 2, 3}"
       )
@@ -973,7 +976,8 @@ defmodule Kernel.ParserTest do
         [
           "nofile:1:14:",
           "unexpected reserved word:",
-          "HINT: the \"<<\" on line 1 is missing terminator \">>\""
+          "└ unclosed delimiter",
+          "└ mismatched closing delimiter"
         ],
         ~c"<<1, 2, 3, 4 end"
       )
@@ -984,7 +988,8 @@ defmodule Kernel.ParserTest do
         [
           "nofile:1:17:",
           "unexpected token:",
-          "HINT: the \"do\" on line 1 is missing terminator \"end\""
+          "└ unclosed delimiter",
+          "└ mismatched closing delimiter"
         ],
         ~c"\"foo\#{case 1 do )}bar\""
       )
@@ -993,7 +998,8 @@ defmodule Kernel.ParserTest do
         [
           "nofile:8:3:",
           "unexpected token: )",
-          "HINT: the \"do\" on line 3 is missing terminator \"end\""
+          "└ unclosed delimiter",
+          "└ mismatched closing delimiter"
         ],
         ~c"""
         defmodule MyApp do
diff --git a/lib/iex/test/iex/interaction_test.exs b/lib/iex/test/iex/interaction_test.exs
index e205c87adfb..3f4de7ae5e3 100644
--- a/lib/iex/test/iex/interaction_test.exs
+++ b/lib/iex/test/iex/interaction_test.exs
@@ -49,7 +49,8 @@ defmodule IEx.InteractionTest do
     assert output =~ "unexpected token: )"
     assert output =~ "iex:1:12"
     assert output =~ "if true do ) false end"
-    assert output =~ ~s/HINT: the "do" on line 1 is missing terminator "end"/
+    assert output =~ "└ unclosed delimiter"
+    assert output =~ "└ mismatched closing delimiter"
   end
 
   test "multiple vars" do

From 7c277d083c555644b8ca6389186c1bb6252da9d1 Mon Sep 17 00:00:00 2001
From: Nathan Long <him@nathanmlong.com>
Date: Fri, 15 Sep 2023 13:22:58 -0400
Subject: [PATCH 06/20] Clarify that registration is only on the local node
 (#12934)

---
 lib/elixir/lib/process.ex | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/elixir/lib/process.ex b/lib/elixir/lib/process.ex
index 522cc7d63d8..e1968538ac2 100644
--- a/lib/elixir/lib/process.ex
+++ b/lib/elixir/lib/process.ex
@@ -651,7 +651,7 @@ defmodule Process do
   defdelegate unlink(pid_or_port), to: :erlang
 
   @doc """
-  Registers the given `pid_or_port` under the given `name`.
+  Registers the given `pid_or_port` under the given `name` on the local node.
 
   `name` must be an atom and can then be used instead of the
   PID/port identifier when sending messages with `Kernel.send/2`.

From 8e6c898fca7829e5f4a47b48838d8267ddef5f74 Mon Sep 17 00:00:00 2001
From: Jean Klingler <sabiwara@gmail.com>
Date: Sat, 16 Sep 2023 16:55:53 +0900
Subject: [PATCH 07/20] Remove duplicate word in guide (#12936)

---
 lib/elixir/pages/getting-started/keywords-and-maps.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/elixir/pages/getting-started/keywords-and-maps.md b/lib/elixir/pages/getting-started/keywords-and-maps.md
index fae2b8cd251..e36c773dcd4 100644
--- a/lib/elixir/pages/getting-started/keywords-and-maps.md
+++ b/lib/elixir/pages/getting-started/keywords-and-maps.md
@@ -190,7 +190,7 @@ iex> map = %{:name => "John", :age => 23}
 
 As you can see from the printed result above, Elixir also allows you to write maps of atom keys using the same `key: value` syntax as keyword lists.
 
-When the keys are atoms, in particular when working with maps of predefined keys, we can also also access them using the `map.key` syntax:
+When the keys are atoms, in particular when working with maps of predefined keys, we can also access them using the `map.key` syntax:
 
 ```elixir
 iex> map = %{name: "John", age: 23}

From ce6c16016c221ef2c858800d43b79163713c0f8a Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jos=C3=A9=20Valim?= <jose.valim@dashbit.co>
Date: Sat, 16 Sep 2023 10:42:25 +0200
Subject: [PATCH 08/20] Revert "Add `<|>` to the list of parsable but unused
 operators (#12932)" (#12937)

This reverts commit a4956eee407d2d744775d95afbc70d42f369db8f.
---
 lib/elixir/pages/references/operators.md | 45 ++++++++++++------------
 1 file changed, 22 insertions(+), 23 deletions(-)

diff --git a/lib/elixir/pages/references/operators.md b/lib/elixir/pages/references/operators.md
index 608abe16d44..41642f0a9f5 100644
--- a/lib/elixir/pages/references/operators.md
+++ b/lib/elixir/pages/references/operators.md
@@ -6,28 +6,28 @@ This document is a complete reference of operators in Elixir, how they are parse
 
 The following is a list of all operators that Elixir is capable of parsing, ordered from higher to lower precedence, alongside their associativity:
 
-Operator                                             | Associativity
----------------------------------------------------- | -------------
-`@`                                                  | Unary
-`.`                                                  | Left
-`+` `-` `!` `^` `not`                                | Unary
-`**`                                                 | Left
-`*` `/`                                              | Left
-`+` `-`                                              | Left
-`++` `--` `+++` `---` `..` `<>`                      | Right
-`in` `not in`                                        | Left
-`\|>` `<\|>` `<<<` `>>>` `<<~` `~>>` `<~` `~>` `<~>` | Left
-`<` `>` `<=` `>=`                                    | Left
-`==` `!=` `=~` `===` `!==`                           | Left
-`&&` `&&&` `and`                                     | Left
-`\|\|` `\|\|\|` `or`                                 | Left
-`=`                                                  | Right
-`&`                                                  | Unary
-`=>` (valid only inside `%{}`)                       | Right
-`\|`                                                 | Right
-`::`                                                 | Right
-`when`                                               | Right
-`<-` `\\`                                            | Left
+Operator                                       | Associativity
+---------------------------------------------- | -------------
+`@`                                            | Unary
+`.`                                            | Left
+`+` `-` `!` `^` `not`                          | Unary
+`**`                                           | Left
+`*` `/`                                        | Left
+`+` `-`                                        | Left
+`++` `--` `+++` `---` `..` `<>`                | Right
+`in` `not in`                                  | Left
+`\|>` `<<<` `>>>` `<<~` `~>>` `<~` `~>` `<~>`  | Left
+`<` `>` `<=` `>=`                              | Left
+`==` `!=` `=~` `===` `!==`                     | Left
+`&&` `&&&` `and`                               | Left
+`\|\|` `\|\|\|` `or`                           | Left
+`=`                                            | Right
+`&`                                            | Unary
+`=>` (valid only inside `%{}`)                 | Right
+`\|`                                           | Right
+`::`                                           | Right
+`when`                                         | Right
+`<-` `\\`                                      | Left
 
 ## General operators
 
@@ -129,7 +129,6 @@ The following is a table of all the operators that Elixir is capable of parsing,
   * `<~`
   * `~>`
   * `<~>`
-  * `<|>`
   * `+++`
   * `---`
 

From f1594048a5f13d12a916f6e88452b1e553656d18 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Vin=C3=ADcius=20M=C3=BCller?= <vinigm.nho@gmail.com>
Date: Sat, 16 Sep 2023 08:59:59 +0000
Subject: [PATCH 09/20] Unicode support for mismatched delimiter errors
 (#12935)

---
 lib/elixir/src/elixir_errors.erl              |  2 +-
 .../test/elixir/kernel/diagnostics_test.exs   | 77 +++++++++++++++++++
 2 files changed, 78 insertions(+), 1 deletion(-)

diff --git a/lib/elixir/src/elixir_errors.erl b/lib/elixir/src/elixir_errors.erl
index 59135b1503d..0b4d9e98a0e 100644
--- a/lib/elixir/src/elixir_errors.erl
+++ b/lib/elixir/src/elixir_errors.erl
@@ -400,7 +400,7 @@ parse_erl_term(Term) ->
 
 raise_mismatched_delimiter(Location, File, Input, Message) ->
   {InputString, _, _} = Input,
-  InputBinary = iolist_to_binary(InputString),
+  InputBinary = elixir_utils:characters_to_binary(InputString),
   raise('Elixir.MismatchedDelimiterError', Message,  [{file, File}, {snippet, InputBinary} | Location]).
 
 raise_reserved(Location, File, Input, Keyword) ->
diff --git a/lib/elixir/test/elixir/kernel/diagnostics_test.exs b/lib/elixir/test/elixir/kernel/diagnostics_test.exs
index 4dc230f2de2..57719f2d3d9 100644
--- a/lib/elixir/test/elixir/kernel/diagnostics_test.exs
+++ b/lib/elixir/test/elixir/kernel/diagnostics_test.exs
@@ -11,6 +11,27 @@ defmodule Kernel.DiagnosticsTest do
   end
 
   describe "mismatched delimiter" do
+    test "same line - handles unicode input" do
+      output =
+        capture_raise(
+          """
+          [1, 2, 3, 4, 5, 6) <- 😎
+          """,
+          MismatchedDelimiterError
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:1:18:
+                 error: unexpected token: )
+                 │
+               1 │ [1, 2, 3, 4, 5, 6) <- 😎
+                 │ │                └ mismatched closing delimiter (expected "]")
+                 │ └ unclosed delimiter
+                 │
+                 └─ nofile:1:18\
+             """
+    end
+
     test "same line" do
       output =
         capture_raise(
@@ -107,6 +128,31 @@ defmodule Kernel.DiagnosticsTest do
              """
     end
 
+    test "line range - handles unicode input" do
+      output =
+        capture_raise(
+          """
+          defmodule A do
+            IO.inspect(2 + 2)
+          ) <- 😎
+          """,
+          MismatchedDelimiterError
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:3:1:
+                 error: unexpected token: )
+                 │
+               1 │ defmodule A do
+                 │             └ unclosed delimiter
+               2 │   IO.inspect(2 + 2)
+               3 │ ) <- 😎
+                 │ └ mismatched closing delimiter (expected "end")
+                 │
+                 └─ nofile:3:1\
+             """
+    end
+
     test "trim inbetween lines if too many" do
       output =
         capture_raise(
@@ -138,6 +184,37 @@ defmodule Kernel.DiagnosticsTest do
              """
     end
 
+    test "trimmed line range - handles unicode input" do
+      output =
+        capture_raise(
+          """
+          [ :a,
+            :b,
+            :c,
+            :d,
+            :e,
+            :f,
+            :g,
+            :h
+          ) <- 😎
+          """,
+          MismatchedDelimiterError
+        )
+
+      assert output == """
+             ** (MismatchedDelimiterError) mismatched delimiter found on nofile:9:1:
+                 error: unexpected token: )
+                 │
+               1 │ [ :a,
+                 │ └ unclosed delimiter
+              ...
+               9 │ ) <- 😎
+                 │ └ mismatched closing delimiter (expected "]")
+                 │
+                 └─ nofile:9:1\
+             """
+    end
+
     test "pads according to line number digits" do
       output =
         capture_raise(

From f017e109f5a77d8ce32df64243e794627844e6f8 Mon Sep 17 00:00:00 2001
From: Lucas Francisco da Matta Vegi <lucasvegi@gmail.com>
Date: Sat, 16 Sep 2023 06:29:58 -0300
Subject: [PATCH 10/20] "Unrelated multi-clause function" added (Anti-patterns
 documentation) (#12931)

---
 .../anti-patterns/design-anti-patterns.md     | 73 ++++++++++++++++++-
 1 file changed, 72 insertions(+), 1 deletion(-)

diff --git a/lib/elixir/pages/anti-patterns/design-anti-patterns.md b/lib/elixir/pages/anti-patterns/design-anti-patterns.md
index fe29f9c8f8e..5e955ce0327 100644
--- a/lib/elixir/pages/anti-patterns/design-anti-patterns.md
+++ b/lib/elixir/pages/anti-patterns/design-anti-patterns.md
@@ -132,7 +132,78 @@ iex> AlternativeInteger.parse_discard_rest("13")
 
 ## Unrelated multi-clause function
 
-TODO
+#### Problem
+
+Using multi-clause functions in Elixir, to group functions of the same name, is not an anti-pattern in itself. However, due to the great flexibility provided by this programming feature, some developers may abuse the number of guard clauses and pattern matches to group *unrelated* functionality.
+
+#### Example
+
+A frequent example of this usage of multi-clause functions is when developers mix unrelated business logic into the same function definition. Such functions often have generic names or too broad specifications, making it difficult for maintainers and users of said functions to maintain and understand them.
+
+Some developers may use documentation mechanisms such as `@doc` annotations to compensate for poor code readability, however the documentation itself may end-up full of conditionals to describe how the function behaves for each different argument combination.
+
+```elixir
+@doc """
+Updates a struct.
+
+If given a "sharp" product (metal or glass with empty count),
+it will...
+
+If given a blunt product, it will...
+
+If given an animal, it will...
+"""
+def update(%Product{count: nil, material: material})
+    when material in ["metal", "glass"] do
+  # ...
+end
+
+def update(%Product{count: count, material: material})
+    when count > 0 and material not in ["metal", "glass"] do
+  # ...
+end
+
+def update(%Animal{count: 1, skin: skin})
+    when skin in ["fur", "hairy"] do
+  # ...
+end
+```
+
+#### Refactoring
+
+As shown below, a possible solution to this anti-pattern is to break the business rules that are mixed up in a single unrelated multi-clause function in several different simple functions. More precise names make the scope of the function clear. Each function can have a specific `@doc`, describing its behavior and parameters received. While this refactoring sounds simple, it can have a lot of impact on the function's current users, so be careful!
+
+```elixir
+@doc """
+Updates a "sharp" product.
+
+It will...
+"""
+def update_sharp_product(%Product{count: nil, material: material})
+    when material in ["metal", "glass"] do
+  # ...
+end
+
+@doc """
+Updates a "blunt" product.
+
+It will...
+"""
+def update_blunt_product(%Product{count: count, material: material})
+    when count > 0 and material not in ["metal", "glass"] do
+  # ...
+end
+
+@doc """
+Updates an animal.
+
+It will...
+"""
+def update_animal(%Animal{count: 1, skin: skin})
+    when skin in ["fur", "hairy"] do
+  # ...
+end
+```
 
 ## Feature envy
 

From 19451c72e9f36b45901283783215f4f75dfe898a Mon Sep 17 00:00:00 2001
From: Christoph Schmatzler <christoph@medium.place>
Date: Sat, 16 Sep 2023 15:48:44 +0200
Subject: [PATCH 11/20] docs: fix reference to `DateTime.time_zone` (#12938)

---
 lib/elixir/lib/calendar/naive_datetime.ex | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/elixir/lib/calendar/naive_datetime.ex b/lib/elixir/lib/calendar/naive_datetime.ex
index a23f320b64e..fe6674679a3 100644
--- a/lib/elixir/lib/calendar/naive_datetime.ex
+++ b/lib/elixir/lib/calendar/naive_datetime.ex
@@ -1224,7 +1224,7 @@ defmodule NaiveDateTime do
 
       datetime
       |> NaiveDateTime.beginning_of_day()
-      |> DateTime.from_naive(datetime.timezone)
+      |> DateTime.from_naive(datetime.time_zone)
 
   Note that the beginning of the day may not exist or be ambiguous
   in a given timezone, so you must handle those cases accordingly.
@@ -1251,7 +1251,7 @@ defmodule NaiveDateTime do
 
       datetime
       |> NaiveDateTime.end_of_day()
-      |> DateTime.from_naive(datetime.timezone)
+      |> DateTime.from_naive(datetime.time_zone)
 
   Note that the end of the day may not exist or be ambiguous
   in a given timezone, so you must handle those cases accordingly.

From abbc3924be6c09a8d79e93a8c1609cb84e858c94 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jos=C3=A9=20Valim?= <jose.valim@dashbit.co>
Date: Sat, 16 Sep 2023 19:15:00 +0200
Subject: [PATCH 12/20] Add instructions on how to run examples, closes #12939

---
 lib/ex_unit/examples/difference.exs  | 1 +
 lib/ex_unit/examples/one_of_each.exs | 1 +
 2 files changed, 2 insertions(+)

diff --git a/lib/ex_unit/examples/difference.exs b/lib/ex_unit/examples/difference.exs
index 20c5afdab48..14e0ed0343d 100644
--- a/lib/ex_unit/examples/difference.exs
+++ b/lib/ex_unit/examples/difference.exs
@@ -1,3 +1,4 @@
+# Run it from root as: make compile && bin/elixir lib/ex_unit/examples/difference.exs
 ExUnit.start(seed: 0)
 
 defmodule Difference do
diff --git a/lib/ex_unit/examples/one_of_each.exs b/lib/ex_unit/examples/one_of_each.exs
index 0a32ddf0c79..e8ff932305b 100644
--- a/lib/ex_unit/examples/one_of_each.exs
+++ b/lib/ex_unit/examples/one_of_each.exs
@@ -1,3 +1,4 @@
+# Run it from root as: make compile && bin/elixir lib/ex_unit/examples/one_of_each.exs
 ExUnit.start(seed: 0)
 
 defmodule TestOneOfEach do

From 10077bdb5ec564313462ae99220605fa26ccf122 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Vin=C3=ADcius=20M=C3=BCller?= <vinigm.nho@gmail.com>
Date: Sat, 16 Sep 2023 18:11:26 +0000
Subject: [PATCH 13/20] Add span to unused/undefined variable diagnostics
 (#12940)

---
 lib/elixir/src/elixir_env.erl                 | 13 +++-
 lib/elixir/src/elixir_errors.erl              |  4 +-
 lib/elixir/src/elixir_expand.erl              |  5 +-
 .../test/elixir/kernel/diagnostics_test.exs   | 64 ++++++++++++++++++-
 4 files changed, 79 insertions(+), 7 deletions(-)

diff --git a/lib/elixir/src/elixir_env.erl b/lib/elixir/src/elixir_env.erl
index 56349b147c3..7bb93f47bbe 100644
--- a/lib/elixir/src/elixir_env.erl
+++ b/lib/elixir/src/elixir_env.erl
@@ -2,7 +2,7 @@
 -include("elixir.hrl").
 -export([
   new/0, to_caller/1, with_vars/2, reset_vars/1, env_to_ex/1, set_prematch_from_config/1,
-  reset_unused_vars/1, check_unused_vars/2, merge_and_check_unused_vars/3,
+  reset_unused_vars/1, check_unused_vars/2, merge_and_check_unused_vars/3, calculate_span/2,
   trace/2, format_error/1,
   reset_read/2, prepare_write/1, close_write/2
 ]).
@@ -85,10 +85,19 @@ reset_unused_vars(#elixir_ex{unused={_Unused, Version}} = S) ->
   S#elixir_ex{unused={#{}, Version}}.
 
 check_unused_vars(#elixir_ex{unused={Unused, _Version}}, E) ->
-  [elixir_errors:file_warn(Meta, E, ?MODULE, {unused_var, Name, Overridden}) ||
+  [elixir_errors:file_warn(calculate_span(Meta, Name), E, ?MODULE, {unused_var, Name, Overridden}) ||
     {{{Name, nil}, _}, {Meta, Overridden}} <- maps:to_list(Unused), is_unused_var(Name)],
   E.
 
+calculate_span(Meta, Name) ->
+  case lists:keyfind(column, 1, Meta) of
+    {column, Column} ->
+      [{span, {?line(Meta), Column + string:length(atom_to_binary(Name))}} | Meta];
+
+    _ ->
+      Meta
+  end.
+
 merge_and_check_unused_vars(S, #elixir_ex{vars={Read, Write}, unused={Unused, _Version}}, E) ->
   #elixir_ex{unused={ClauseUnused, Version}} = S,
   NewUnused = merge_and_check_unused_vars(Read, Unused, ClauseUnused, E),
diff --git a/lib/elixir/src/elixir_errors.erl b/lib/elixir/src/elixir_errors.erl
index 0b4d9e98a0e..7f450b3aa5d 100644
--- a/lib/elixir/src/elixir_errors.erl
+++ b/lib/elixir/src/elixir_errors.erl
@@ -251,7 +251,7 @@ file_warn(Meta, E, Module, Desc) when is_list(Meta) ->
     false ->
       {EnvPosition, EnvFile, EnvStacktrace} = env_format(Meta, E),
       Message = Module:format_error(Desc),
-      emit_diagnostic(warning, EnvPosition, EnvFile, Message, EnvStacktrace, [{read_snippet, true}])
+      emit_diagnostic(warning, EnvPosition, EnvFile, Message, EnvStacktrace, [{read_snippet, true} | Meta])
   end.
 
 -spec file_error(list(), binary() | #{file := binary(), _ => _}, module(), any()) -> no_return().
@@ -284,7 +284,7 @@ function_error(Meta, Env, Module, Desc) ->
 print_error(Meta, Env, Module, Desc) ->
   {EnvPosition, EnvFile, EnvStacktrace} = env_format(Meta, Env),
   Message = Module:format_error(Desc),
-  emit_diagnostic(error, EnvPosition, EnvFile, Message, EnvStacktrace, [{read_snippet, true}]),
+  emit_diagnostic(error, EnvPosition, EnvFile, Message, EnvStacktrace, [{read_snippet, true} | Meta]),
   ok.
 
 %% Compilation error.
diff --git a/lib/elixir/src/elixir_expand.erl b/lib/elixir/src/elixir_expand.erl
index a4cd14b9d43..62b1f7c5583 100644
--- a/lib/elixir/src/elixir_expand.erl
+++ b/lib/elixir/src/elixir_expand.erl
@@ -380,8 +380,9 @@ expand({Name, Meta, Kind}, S, E) when is_atom(Name), is_atom(Kind) ->
           {{Name, Meta, Kind}, S, E};
 
         _ ->
-          function_error(Meta, E, ?MODULE, {undefined_var, Name, Kind}),
-          {{Name, Meta, Kind}, S, E}
+          SpanMeta =  elixir_env:calculate_span(Meta, Name),
+          function_error(SpanMeta, E, ?MODULE, {undefined_var, Name, Kind}),
+          {{Name, SpanMeta, Kind}, S, E}
       end
   end;
 
diff --git a/lib/elixir/test/elixir/kernel/diagnostics_test.exs b/lib/elixir/test/elixir/kernel/diagnostics_test.exs
index 57719f2d3d9..e074475d5f3 100644
--- a/lib/elixir/test/elixir/kernel/diagnostics_test.exs
+++ b/lib/elixir/test/elixir/kernel/diagnostics_test.exs
@@ -827,6 +827,68 @@ defmodule Kernel.DiagnosticsTest do
       purge(Sample)
     end
 
+    @tag :tmp_dir
+    test "shows span for unused variables", %{tmp_dir: tmp_dir} do
+      path = make_relative_tmp(tmp_dir, "error_line_column.ex")
+
+      source = """
+      defmodule Sample do
+        @file "#{path}"
+
+        def foo(unused_param) do
+          :constant
+        end
+      end
+      """
+
+      File.write!(path, source)
+
+      expected = """
+          warning: variable "unused_param" is unused (if the variable is not meant to be used, prefix it with an underscore)
+          │
+        4 │   def foo(unused_param) do
+          │           ~~~~~~~~~~~~
+          │
+          └─ #{path}:4:11: Sample.foo/1
+
+      """
+
+      assert capture_eval(source) == expected
+    after
+      purge(Sample)
+    end
+
+    @tag :tmp_dir
+    test "shows span for undefined variables", %{tmp_dir: tmp_dir} do
+      path = make_relative_tmp(tmp_dir, "undefined_variable_span.ex")
+
+      source = """
+      defmodule Sample do
+        @file "#{path}"
+
+        def foo(a) do
+          a - unknown_var
+        end
+      end
+      """
+
+      File.write!(path, source)
+
+      expected = """
+          error: undefined variable "unknown_var"
+          │
+        5 │     a - unknown_var
+          │         ^^^^^^^^^^^
+          │
+          └─ #{path}:5:9: Sample.foo/1
+
+      """
+
+      assert capture_compile(source) == expected
+    after
+      purge(Sample)
+    end
+
     @tag :tmp_dir
     test "line + column", %{tmp_dir: tmp_dir} do
       path = make_relative_tmp(tmp_dir, "error_line_column.ex")
@@ -847,7 +909,7 @@ defmodule Kernel.DiagnosticsTest do
           error: undefined variable "bar"
           │
         5 │     IO.puts(bar)
-          │             ^
+          │             ^^^
           │
           └─ #{path}:5:13: Sample.foo/0
 

From 32690dd7d1fb4caa94b0f0cbade5fa63dca2234c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jos=C3=A9=20Valim?= <jose.valim@dashbit.co>
Date: Sat, 16 Sep 2023 20:30:04 +0200
Subject: [PATCH 14/20] Read snippets as binaries

---
 lib/elixir/lib/module/parallel_checker.ex |  3 +-
 lib/elixir/src/elixir_errors.erl          | 42 +++++++++++++----------
 2 files changed, 24 insertions(+), 21 deletions(-)

diff --git a/lib/elixir/lib/module/parallel_checker.ex b/lib/elixir/lib/module/parallel_checker.ex
index e6fd474075e..27f2ee00a68 100644
--- a/lib/elixir/lib/module/parallel_checker.ex
+++ b/lib/elixir/lib/module/parallel_checker.ex
@@ -155,8 +155,7 @@ defmodule Module.ParallelChecker do
 
     case :erlang.get(:elixir_code_diagnostics) do
       :undefined -> :ok
-      {tail, true} -> :erlang.put(:elixir_code_diagnostics, {diagnostics ++ tail, true})
-      {tail, false} -> :erlang.put(:elixir_code_diagnostics, {diagnostics ++ tail, false})
+      {tail, log?} -> :erlang.put(:elixir_code_diagnostics, {diagnostics ++ tail, log?})
     end
 
     diagnostics
diff --git a/lib/elixir/src/elixir_errors.erl b/lib/elixir/src/elixir_errors.erl
index 7f450b3aa5d..2bc706df1c3 100644
--- a/lib/elixir/src/elixir_errors.erl
+++ b/lib/elixir/src/elixir_errors.erl
@@ -50,13 +50,32 @@ get_span(#{span := Span}) -> Span.
 
 get_snippet(nil, _Position) ->
   nil;
+get_snippet(<<"nofile">>, _Position) ->
+  nil;
 get_snippet(File, Position) ->
-  Line = extract_line(Position),
-  case filelib:is_regular(File) of
-    true -> get_file_line(File, Line);
-    false -> nil
+  LineNumber = extract_line(Position),
+  get_file_line(File, LineNumber).
+
+get_file_line(_, 0) -> nil;
+get_file_line(File, LineNumber) ->
+  case file:open(File, [read, binary]) of
+    {ok, IoDevice} ->
+      Line = traverse_file_line(IoDevice, LineNumber),
+      ok = file:close(IoDevice),
+      Line;
+    {error, _} ->
+      nil
   end.
 
+traverse_file_line(IoDevice, 1) ->
+  case file:read_line(IoDevice) of
+    {ok, Line} -> binary:replace(Line, <<"\n">>, <<>>);
+    _ -> nil
+  end;
+traverse_file_line(IoDevice, N) ->
+  file:read_line(IoDevice),
+  traverse_file_line(IoDevice, N - 1).
+
 print_diagnostic(#{severity := Severity, message := M, stacktrace := Stacktrace, position := P, file := F} = Diagnostic, ReadSnippet) ->
   Snippet =
     case ReadSnippet of
@@ -112,21 +131,6 @@ extract_line(L) -> L.
 extract_column({_, C}) -> C;
 extract_column(_) -> nil.
 
-get_file_line(_, 0) -> nil;
-get_file_line(File, LineNumber) ->
-  {ok, IoDevice} = file:open(File, [read, {encoding, unicode}]),
-  Line = do_get_file_line(IoDevice, LineNumber),
-  NoNewline = binary:replace(Line, <<"\n">>, <<>>),
-  ok = file:close(IoDevice),
-  NoNewline.
-
-do_get_file_line(IoDevice, 1) ->
-  Line = io:get_line(IoDevice, ""),
-  unicode:characters_to_binary(Line);
-do_get_file_line(IoDevice, N) ->
-  io:get_line(IoDevice, ""),
-  do_get_file_line(IoDevice, N -1).
-
 %% Format snippets
 %% "Snippet" here refers to the source code line where the diagnostic/error occured
 

From df114285ce156a3ba0f1dac3c123c2419fd98e7e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jos=C3=A9=20Valim?= <jose.valim@dashbit.co>
Date: Sun, 17 Sep 2023 10:13:45 +0200
Subject: [PATCH 15/20] Split adjust indent into text and code

---
 lib/ex_unit/lib/ex_unit/doc_test.ex | 31 +++++++++++++----------------
 1 file changed, 14 insertions(+), 17 deletions(-)

diff --git a/lib/ex_unit/lib/ex_unit/doc_test.ex b/lib/ex_unit/lib/ex_unit/doc_test.ex
index 4b6d8a34e44..8224d4fe102 100644
--- a/lib/ex_unit/lib/ex_unit/doc_test.ex
+++ b/lib/ex_unit/lib/ex_unit/doc_test.ex
@@ -610,25 +610,29 @@ defmodule ExUnit.DocTest do
   @fences ["```", "~~~"]
 
   defp adjust_indent(lines, line_no, module) do
-    adjust_indent(:text, lines, line_no, [], 0, module)
+    adjust_text(lines, line_no, [], module)
   end
 
-  defp adjust_indent(_kind, [], line_no, adjusted_lines, _indent, _module) do
+  defp adjust_text([], line_no, adjusted_lines, _module) do
     {Enum.reverse(adjusted_lines), line_no - 1}
   end
 
-  defp adjust_indent(:text, [line | rest], line_no, adjusted_lines, indent, module) do
+  defp adjust_text([line | rest], line_no, adjusted_lines, module) do
     case String.starts_with?(String.trim_leading(line), @iex_prompt) do
       true ->
-        line_indent = get_indent(line, indent)
-        adjust_indent(:prompt, [line | rest], line_no, adjusted_lines, line_indent, module)
+        {indent, _len} = :binary.match(line, "iex")
+        adjust_code(:prompt, [line | rest], line_no, adjusted_lines, indent, module)
 
       false ->
-        adjust_indent(:text, rest, line_no + 1, adjusted_lines, indent, module)
+        adjust_text(rest, line_no + 1, adjusted_lines, module)
     end
   end
 
-  defp adjust_indent(kind, [line | rest], line_no, adjusted_lines, indent, module) do
+  defp adjust_code(_kind, [], line_no, adjusted_lines, _indent, _module) do
+    {Enum.reverse(adjusted_lines), line_no - 1}
+  end
+
+  defp adjust_code(kind, [line | rest], line_no, adjusted_lines, indent, module) do
     stripped_line = strip_indent(line, indent)
     trimmed_line = String.trim_leading(line)
     done? = stripped_line == "" or String.starts_with?(stripped_line, @fences)
@@ -659,23 +663,16 @@ defmodule ExUnit.DocTest do
     cond do
       done? ->
         adjusted_lines = [{"", line_no} | adjusted_lines]
-        adjust_indent(:text, rest, line_no + 1, adjusted_lines, 0, module)
+        adjust_text(rest, line_no + 1, adjusted_lines, module)
 
       kind == :prompt or String.starts_with?(trimmed_line, @iex_prompt) or
           (kind == :maybe_prompt and String.starts_with?(trimmed_line, @dot_prompt)) ->
         line = {adjust_prompt(stripped_line, line_no, module), line_no}
-        adjust_indent(:maybe_prompt, rest, line_no + 1, [line | adjusted_lines], indent, module)
+        adjust_code(:maybe_prompt, rest, line_no + 1, [line | adjusted_lines], indent, module)
 
       true ->
         adjusted_lines = [{stripped_line, line_no} | adjusted_lines]
-        adjust_indent(:code, rest, line_no + 1, adjusted_lines, indent, module)
-    end
-  end
-
-  defp get_indent(line, current_indent) do
-    case :binary.match(line, "iex") do
-      {pos, _len} -> pos
-      :nomatch -> current_indent
+        adjust_code(:code, rest, line_no + 1, adjusted_lines, indent, module)
     end
   end
 

From 77abd0606f22c3217f36f7ddb295784073dc8356 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jos=C3=A9=20Valim?= <jose.valim@dashbit.co>
Date: Sun, 17 Sep 2023 11:29:01 +0200
Subject: [PATCH 16/20] Have references come last

---
 lib/elixir/scripts/elixir_docs.exs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/elixir/scripts/elixir_docs.exs b/lib/elixir/scripts/elixir_docs.exs
index c85c5e2a4d9..1f7303d91e1 100644
--- a/lib/elixir/scripts/elixir_docs.exs
+++ b/lib/elixir/scripts/elixir_docs.exs
@@ -68,9 +68,9 @@ canonical = System.fetch_env!("CANONICAL")
     "Getting started": ~r"pages/getting-started/.*\.md$",
     Cheatsheets: ~r"pages/cheatsheets/.*\.cheatmd$",
     "Anti-patterns": ~r"pages/anti-patterns/.*\.md$",
-    References: ~r"pages/references/.*\.md$",
     "Meta-programming": ~r"pages/meta-programming/.*\.md$",
-    "Mix & OTP": ~r"pages/mix-and-otp/.*\.md$"
+    "Mix & OTP": ~r"pages/mix-and-otp/.*\.md$",
+    References: ~r"pages/references/.*\.md$"
   ],
   groups_for_functions: [
     Guards: &(&1[:guard] == true)

From cb9de080e50fee94dc3422a540db24cfaf03a044 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jos=C3=A9=20Valim?= <jose.valim@dashbit.co>
Date: Sun, 17 Sep 2023 22:25:33 +0200
Subject: [PATCH 17/20] Strip column information on quote

---
 lib/elixir/src/elixir_quote.erl               |  4 +--
 .../test/elixir/kernel/tracers_test.exs       | 29 ++++++++++++++++++-
 2 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/lib/elixir/src/elixir_quote.erl b/lib/elixir/src/elixir_quote.erl
index b31ac12c867..730dd0e252e 100644
--- a/lib/elixir/src/elixir_quote.erl
+++ b/lib/elixir/src/elixir_quote.erl
@@ -525,9 +525,9 @@ argument_error(Message) ->
 %% Helpers
 
 meta(Meta, Q) ->
-  generated(keep(Meta, Q), Q).
+  generated(keep(keydelete(column, Meta), Q), Q).
 
-generated(Meta, #elixir_quote{generated=true}) -> elixir_utils:generated(Meta);
+generated(Meta, #elixir_quote{generated=true}) -> [{generated, true} | Meta];
 generated(Meta, #elixir_quote{generated=false}) -> Meta.
 
 keep(Meta, #elixir_quote{file=nil, line=Line}) ->
diff --git a/lib/elixir/test/elixir/kernel/tracers_test.exs b/lib/elixir/test/elixir/kernel/tracers_test.exs
index 3116b067388..bf9c5fcd4da 100644
--- a/lib/elixir/test/elixir/kernel/tracers_test.exs
+++ b/lib/elixir/test/elixir/kernel/tracers_test.exs
@@ -3,7 +3,11 @@ Code.require_file("../test_helper.exs", __DIR__)
 defmodule Kernel.TracersTest do
   use ExUnit.Case
 
-  import Code, only: [compile_string: 1]
+  defp compile_string(string) do
+    string
+    |> Code.string_to_quoted!(columns: true)
+    |> Code.compile_quoted()
+  end
 
   def trace(event, %Macro.Env{} = env) do
     send(self(), {event, env})
@@ -204,4 +208,27 @@ defmodule Kernel.TracersTest do
     :code.purge(Sample)
     :code.delete(Sample)
   end
+
+  """
+  # Make sure this module is compiled with column information
+  defmodule MacroWithColumn do
+    defmacro some_macro(list) do
+      quote do
+        Enum.map(unquote(list), fn str -> String.upcase(str) end)
+      end
+    end
+  end
+  """
+  |> Code.string_to_quoted!(columns: true)
+  |> Code.compile_quoted()
+
+  test "traces quoted from macro expansion without column information" do
+    compile_string("""
+    require MacroWithColumn
+    MacroWithColumn.some_macro(["hello", "world", "!"])
+    """)
+
+    assert_receive {{:alias_reference, meta, Enum}, _env}
+    refute meta[:column]
+  end
 end

From 5225b33bab94367d818b557dabb3fff65b3e0660 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jos=C3=A9=20Valim?= <jose.valim@dashbit.co>
Date: Sun, 17 Sep 2023 22:43:07 +0200
Subject: [PATCH 18/20] Add interpolation token metadata

---
 lib/elixir/lib/macro.ex                        | 16 +++++++++++++---
 lib/elixir/src/elixir_parser.yrl               |  8 ++++----
 lib/elixir/test/elixir/kernel/tracers_test.exs | 10 ++++++++++
 3 files changed, 27 insertions(+), 7 deletions(-)

diff --git a/lib/elixir/lib/macro.ex b/lib/elixir/lib/macro.ex
index bb6883e2262..4441e1c8041 100644
--- a/lib/elixir/lib/macro.ex
+++ b/lib/elixir/lib/macro.ex
@@ -181,6 +181,12 @@ defmodule Macro do
       the compiler, each variable is identified by the combination of either
       `name` and `metadata[:counter]`, or `name` and `context`.
 
+    * `:from_brackets` - Used to determine whether a call to `Access.get/3` is from
+      bracket syntax.
+
+    * `:from_interpolation` - Used to determine whether a call to `Access.get/3` is
+      from interpolation.
+
     * `:generated` - Whether the code should be considered as generated by
       the compiler or not. This means the compiler and tools like Dialyzer may not
       emit certain warnings.
@@ -194,26 +200,30 @@ defmodule Macro do
 
     * `:line` - The line number of the AST node.
 
-    * `:from_brackets` - Used to determine whether a call to `Access.get/3` is from
-      bracket syntax or a function call.
-
   The following metadata keys are enabled by `Code.string_to_quoted/2`:
 
     * `:closing` - contains metadata about the closing pair, such as a `}`
       in a tuple or in a map, or such as the closing `)` in a function call
       with parens. The `:closing` does not delimit the end of expression if
       there are `:do` and `:end` metadata  (when `:token_metadata` is true)
+
     * `:column` - the column number of the AST node (when `:columns` is true)
+
     * `:delimiter` - contains the opening delimiter for sigils, strings,
       and charlists as a string (such as `"{"`, `"/"`, `"'"`, and the like)
+
     * `:format` - set to `:keyword` when an atom is defined as a keyword
+
     * `:do` - contains metadata about the `do` location in a function call with
       `do`-`end` blocks (when `:token_metadata` is true)
+
     * `:end` - contains metadata about the `end` location in a function call with
       `do`-`end` blocks (when `:token_metadata` is true)
+
     * `:end_of_expression` - denotes when the end of expression effectively
       happens. Available for all expressions except the last one inside a
       `__block__` (when `:token_metadata` is true)
+
     * `:indentation` - indentation of a sigil heredoc
 
   The following metadata keys are private:
diff --git a/lib/elixir/src/elixir_parser.yrl b/lib/elixir/src/elixir_parser.yrl
index d5341acd666..de678104fd2 100644
--- a/lib/elixir/src/elixir_parser.yrl
+++ b/lib/elixir/src/elixir_parser.yrl
@@ -1019,7 +1019,7 @@ charlist_part({Begin, End, Tokens}) ->
       true -> [{closing, meta_from_location(End)} | Meta];
       false -> Meta
     end,
-  {{'.', Meta, ['Elixir.Kernel', to_string]}, MetaWithExtra, [Form]}.
+  {{'.', Meta, ['Elixir.Kernel', to_string]}, [{from_interpolation, true} | MetaWithExtra], [Form]}.
 
 string_parts(Parts) ->
   [string_part(Part) || Part <- Parts].
@@ -1030,10 +1030,10 @@ string_part({Begin, End, Tokens}) ->
   Meta = meta_from_location(Begin),
   MetaWithExtra =
     case ?token_metadata() of
-      true -> [{closing, meta_from_location(End)} | meta_from_location(Begin)];
-      false -> meta_from_location(Begin)
+      true -> [{closing, meta_from_location(End)} | Meta];
+      false -> Meta
     end,
-  {'::', Meta, [{{'.', Meta, ['Elixir.Kernel', to_string]}, MetaWithExtra, [Form]}, {binary, Meta, nil}]}.
+  {'::', Meta, [{{'.', Meta, ['Elixir.Kernel', to_string]}, [{from_interpolation, true} | MetaWithExtra], [Form]}, {binary, Meta, nil}]}.
 
 string_tokens_parse(Tokens) ->
   case parse(Tokens) of
diff --git a/lib/elixir/test/elixir/kernel/tracers_test.exs b/lib/elixir/test/elixir/kernel/tracers_test.exs
index bf9c5fcd4da..6bc2ccf6592 100644
--- a/lib/elixir/test/elixir/kernel/tracers_test.exs
+++ b/lib/elixir/test/elixir/kernel/tracers_test.exs
@@ -209,6 +209,16 @@ defmodule Kernel.TracersTest do
     :code.delete(Sample)
   end
 
+  test "traces string interpolation" do
+    compile_string("""
+    arg = 1 + 2
+    "foo\#{arg}"
+    """)
+
+    assert_receive {{:remote_macro, meta, Kernel, :to_string, 1}, _env}
+    assert meta[:from_interpolation]
+  end
+
   """
   # Make sure this module is compiled with column information
   defmodule MacroWithColumn do

From f632fc648e4dfdde2a17f71abe9e861b35b06075 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Vin=C3=ADcius=20M=C3=BCller?= <vinigm.nho@gmail.com>
Date: Mon, 18 Sep 2023 07:42:17 +0000
Subject: [PATCH 19/20] Set parser_options[:columns] to true by default
 (#12941)

---
 lib/elixir/src/elixir.erl                     |  2 +-
 .../elixir/module/types/integration_test.exs  | 72 +++++++++----------
 .../test/mix/tasks/compile.elixir_test.exs    |  4 +-
 3 files changed, 39 insertions(+), 39 deletions(-)

diff --git a/lib/elixir/src/elixir.erl b/lib/elixir/src/elixir.erl
index cdd0b7d4c7c..575756fa659 100644
--- a/lib/elixir/src/elixir.erl
+++ b/lib/elixir/src/elixir.erl
@@ -89,7 +89,7 @@ start(_Type, _Args) ->
     {ignore_already_consolidated, false},
     {ignore_module_conflict, false},
     {on_undefined_variable, raise},
-    {parser_options, []},
+    {parser_options, [{columns, true}]},
     {debug_info, true},
     {warnings_as_errors, false},
     {relative_paths, true},
diff --git a/lib/elixir/test/elixir/module/types/integration_test.exs b/lib/elixir/test/elixir/module/types/integration_test.exs
index 3e5bf062b26..59fc5b0b7f9 100644
--- a/lib/elixir/test/elixir/module/types/integration_test.exs
+++ b/lib/elixir/test/elixir/module/types/integration_test.exs
@@ -67,9 +67,9 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         ":not_a_module.no_module/0 is undefined (module :not_a_module is not available or is yet to be defined)",
-        "a.ex:2: A.a/0",
+        "a.ex:2:27: A.a/0",
         ":lists.no_func/0 is undefined or private",
-        "a.ex:3: A.b/0"
+        "a.ex:3:20: A.b/0"
       ]
 
       assert_warnings(files, warnings)
@@ -90,7 +90,7 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "Kernel.behaviour_info/1 is undefined or private",
-        "a.ex:6: A.e/0"
+        "a.ex:6:20: A.e/0"
       ]
 
       assert_warnings(files, warnings)
@@ -125,7 +125,7 @@ defmodule Module.Types.IntegrationTest do
         "List.old_flatten/1 is undefined or private. Did you mean:",
         "* flatten/1",
         "* flatten/2",
-        "a.ex:15: A.flatten2/1"
+        "a.ex:15:32: A.flatten2/1"
       ]
 
       assert_warnings(files, warnings)
@@ -146,9 +146,9 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "A.no_func/0 is undefined or private",
-        "a.ex:2: A.a/0",
+        "a.ex:2:15: A.a/0",
         "A.no_func/1 is undefined or private",
-        "external_source.ex:6: A.c/0"
+        "external_source.ex:6:14: A.c/0"
       ]
 
       assert_warnings(files, warnings)
@@ -170,10 +170,10 @@ defmodule Module.Types.IntegrationTest do
       warnings = [
         "A.a/1 is undefined or private. Did you mean:",
         "* a/0",
-        "a.ex:3: A.b/0",
+        "a.ex:3:15: A.b/0",
         "A.b/1 is undefined or private. Did you mean:",
         "* b/0",
-        "external_source.ex:6: A.c/0"
+        "external_source.ex:6:15: A.c/0"
       ]
 
       assert_warnings(files, warnings)
@@ -195,11 +195,11 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "D.no_module/0 is undefined (module D is not available or is yet to be defined)",
-        "a.ex:2: A.a/0",
+        "a.ex:2:15: A.a/0",
         "E.no_module/0 is undefined (module E is not available or is yet to be defined)",
-        "external_source.ex:5: A.c/0",
+        "external_source.ex:5:15: A.c/0",
         "Io.puts/1 is undefined (module Io is not available or is yet to be defined)",
-        "a.ex:7: A.i/0"
+        "a.ex:7:16: A.i/0"
       ]
 
       assert_warnings(files, warnings)
@@ -219,9 +219,9 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "A.no_func/0 is undefined or private",
-        "a.ex:2: A.a/0",
+        "a.ex:2:14: A.a/0",
         "A.no_func/1 is undefined or private",
-        "external_source.ex:5: A.c/0"
+        "external_source.ex:5:14: A.c/0"
       ]
 
       assert_warnings(files, warnings)
@@ -261,9 +261,9 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "B.no_func/0 is undefined or private",
-        "a.ex:2: A.a/0",
+        "a.ex:2:15: A.a/0",
         "A.no_func/0 is undefined or private",
-        "b.ex:2: B.a/0"
+        "b.ex:2:15: B.a/0"
       ]
 
       assert_warnings(files, warnings)
@@ -286,11 +286,11 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "A2.no_func/0 is undefined (module A2 is not available or is yet to be defined)",
-        "└─ a.ex:8: A.d/0",
-        "└─ external_source.ex:5: A.b/0",
+        "└─ a.ex:8:16: A.d/0",
+        "└─ external_source.ex:5:16: A.b/0",
         "A.no_func/0 is undefined or private",
-        "└─ a.ex:2: A.a/0",
-        "└─ a.ex:7: A.c/0"
+        "└─ a.ex:2:15: A.a/0",
+        "└─ a.ex:7:15: A.c/0"
       ]
 
       assert_warnings(files, warnings)
@@ -313,7 +313,7 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "B.no_func/0 is undefined (module B is not available or is yet to be defined)",
-        "a.ex:7: AProtocol.AImplementation.func/1"
+        "a.ex:7:23: AProtocol.AImplementation.func/1"
       ]
 
       assert_warnings(files, warnings)
@@ -349,7 +349,7 @@ defmodule Module.Types.IntegrationTest do
       warnings = [
         "A.to_list/1 is undefined or private. Did you mean:",
         "* to_charlist/1",
-        "a.ex:7: A.c/1"
+        "a.ex:7:18: A.c/1"
       ]
 
       assert_warnings(files, warnings)
@@ -388,7 +388,7 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "you must require B before invoking the macro B.b/0",
-        "ab.ex:2: A.a/0"
+        "ab.ex:2:17: A.a/0"
       ]
 
       assert_warnings(files, warnings)
@@ -419,12 +419,12 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "MissingModule2.func/1 is undefined (module MissingModule2 is not available or is yet to be defined)",
-        "a.ex:7: A.c/0",
+        "a.ex:7:28: A.c/0",
         "MissingModule3.func/2 is undefined (module MissingModule3 is not available or is yet to be defined)",
-        "a.ex:8: A.d/0",
+        "a.ex:8:28: A.d/0",
         "B.func/3 is undefined or private. Did you mean:",
         "* func/1",
-        "a.ex:11: A.g/0"
+        "a.ex:11:15: A.g/0"
       ]
 
       assert_warnings(files, warnings)
@@ -462,12 +462,12 @@ defmodule Module.Types.IntegrationTest do
 
         warnings = [
           "MissingModule2.func/1 is undefined (module MissingModule2 is not available or is yet to be defined)",
-          "a.ex:7: A.c/0",
+          "a.ex:7:28: A.c/0",
           "MissingModule3.func/2 is undefined (module MissingModule3 is not available or is yet to be defined)",
-          "a.ex:8: A.d/0",
+          "a.ex:8:28: A.d/0",
           "B.func/3 is undefined or private. Did you mean:",
           "* func/1",
-          "a.ex:11: A.g/0"
+          "a.ex:11:15: A.g/0"
         ]
 
         assert_warnings(files, warnings)
@@ -588,7 +588,7 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "A.a/0 is deprecated. oops",
-        "a.ex:3: A.a/0"
+        "a.ex:3:15: A.a/0"
       ]
 
       assert_warnings(files, warnings)
@@ -612,7 +612,7 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "A.a/0 is deprecated. oops",
-        "b.ex:3: B.b/0"
+        "b.ex:3:14: B.b/0"
       ]
 
       assert_warnings(files, warnings)
@@ -638,11 +638,11 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "A.__struct__/0 is deprecated. oops",
-        "└─ a.ex:4: A.match/1",
-        "└─ a.ex:5: A.build/1",
+        "└─ a.ex:4:13: A.match/1",
+        "└─ a.ex:5:23: A.build/1",
         "A.__struct__/0 is deprecated. oops",
-        "└─ b.ex:2: B.match/1",
-        "└─ b.ex:3: B.build/1"
+        "└─ b.ex:2:13: B.match/1",
+        "└─ b.ex:3:23: B.build/1"
       ]
 
       assert_warnings(files, warnings)
@@ -666,7 +666,7 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "A.a/0 is deprecated. oops",
-        "b.ex:3: B (module)"
+        "b.ex:3:5: B (module)"
       ]
 
       assert_warnings(files, warnings)
@@ -690,7 +690,7 @@ defmodule Module.Types.IntegrationTest do
 
       warnings = [
         "A.a/0 is deprecated. oops",
-        "b.ex:3: B.b/0"
+        "b.ex:3:16: B.b/0"
       ]
 
       assert_warnings(files, warnings)
diff --git a/lib/mix/test/mix/tasks/compile.elixir_test.exs b/lib/mix/test/mix/tasks/compile.elixir_test.exs
index e2bb55930a8..038fd8039fa 100644
--- a/lib/mix/test/mix/tasks/compile.elixir_test.exs
+++ b/lib/mix/test/mix/tasks/compile.elixir_test.exs
@@ -1416,7 +1416,7 @@ defmodule Mix.Tasks.Compile.ElixirTest do
         assert %Diagnostic{
                  file: ^file,
                  severity: :warning,
-                 position: 2,
+                 position: {2, 13},
                  compiler_name: "Elixir",
                  message: ^message
                } = diagnostic
@@ -1430,7 +1430,7 @@ defmodule Mix.Tasks.Compile.ElixirTest do
         assert %Diagnostic{
                  file: ^file,
                  severity: :warning,
-                 position: 2,
+                 position: {2, 13},
                  compiler_name: "Elixir",
                  message: ^message
                } = diagnostic

From 6510f25e30ee764d35b081a028e8c34993d8b493 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Jos=C3=A9=20Valim?= <jose.valim@dashbit.co>
Date: Mon, 18 Sep 2023 09:44:10 +0200
Subject: [PATCH 20/20] Improve parsing and ast metadata docs

---
 lib/elixir/lib/code.ex  | 2 +-
 lib/elixir/lib/macro.ex | 6 ++++--
 2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/lib/elixir/lib/code.ex b/lib/elixir/lib/code.ex
index 163de0b296e..09099667a5c 100644
--- a/lib/elixir/lib/code.ex
+++ b/lib/elixir/lib/code.ex
@@ -1595,7 +1595,7 @@ defmodule Code do
       to the parser when compiling files. It accepts the same options as
       `string_to_quoted/2` (except by the options that change the AST itself).
       This can be used in combination with the tracer to retrieve localized
-      information about events happening during compilation. Defaults to `[]`.
+      information about events happening during compilation. Defaults to `[columns: true]`.
       This option only affects code compilation functions, such as `compile_string/2`
       and `compile_file/2` but not `string_to_quoted/2` and friends, as the
       latter is used for other purposes beyond compilation.
diff --git a/lib/elixir/lib/macro.ex b/lib/elixir/lib/macro.ex
index 4441e1c8041..1c091ec477c 100644
--- a/lib/elixir/lib/macro.ex
+++ b/lib/elixir/lib/macro.ex
@@ -198,7 +198,8 @@ defmodule Macro do
     * `:keep` - Used by `quote/2` with the option `location: :keep` to annotate
       the file and the line number of the quoted source.
 
-    * `:line` - The line number of the AST node.
+    * `:line` - The line number of the AST node. Note line information is discarded
+      from quoted code but can be enabled back via the `:line` option.
 
   The following metadata keys are enabled by `Code.string_to_quoted/2`:
 
@@ -207,7 +208,8 @@ defmodule Macro do
       with parens. The `:closing` does not delimit the end of expression if
       there are `:do` and `:end` metadata  (when `:token_metadata` is true)
 
-    * `:column` - the column number of the AST node (when `:columns` is true)
+    * `:column` - the column number of the AST node (when `:columns` is true).
+      Note column information is always discarded from quoted code.
 
     * `:delimiter` - contains the opening delimiter for sigils, strings,
       and charlists as a string (such as `"{"`, `"/"`, `"'"`, and the like)