Begin work on lexing and parsing

[Phate6660]

Currently the REPL is very basic.
It uses basic pattern matching features on larger variable contents.
It's about to time to implement a lexer and parser to more accurately scan the line being entered.
And once I get lexing and parsing set up I can get going on the shell scripting side of things.

[Phate6660]

Current thoughts on this right now:

• Implement using serde, which seems to be the de facto standard for this stuff in Rust [WIP -- maybe]
• Implement using a different crate
• Implement lexing and parsing by hand [will most likely do this*]

* Matches up with what I plan for the shell in the future. By the time I feel this shell is "ready", I want to have any and all deps written from scratch.
I'm currently only using pre-made crates right now to get the shell on it's feet so to speak.

[Phate6660]

@TheOnAndOnlyZenomat This is what I wanted to bring up.
Honestly, only supporting one set of quotes right now is fine,
because I plan to kill multiple birds with one stone with the PRs that will accompany the issue.
Tokenizing, lexing, and parsing the command line string will allow us to:

• Start developing shell scripting capabilities
• Recursively support all kinds of syntactic stuff for the shell we create
• Better handling of any command line errors that may arise
• And modify the syntax of the shell in a centralized and easier way

[TheOnAndOnlyZenomat]

It sounds sensible to implement lexing and parsing.
I basically know nothing about lexing and parsing, so I'll read up on that.
All I think I know about that topic, is, that it is related to creating a programming language, so way over my head, but could be interesting to get into.

[TheOnAndOnlyZenomat]

• Matches up with what I plan for the shell in the future. By the time I feel this shell is "ready", I want to have any and all deps written from scratch. I'm currently only using pre-made crates right now to get the shell on it's feet so to speak.

This is gonna be a hard task, I mean I totally vibe with the idea and I really like it, but it's gonna be hard^^ But let's do it

[TheOnAndOnlyZenomat]

Do you by chance have a good resource to start to read about that topic?

[Phate6660]

Nah. But I can look around for some, and I do have previous experience
with tokenizing, lexing, and parsing in previous projects.

[TheOnAndOnlyZenomat]

Alright, sounds good. Do you have a link to such an example project, so that I can take a look at it?

[Phate6660]

Yep! https://github.com/Phate6660/bcalc is a major one!

[Phate6660]

By the way. I have started work on this.
Once I have a good base set up (which should be tomorrow (preferably) or the day after),
I'll make a branch that'll be tracked.
I'm currently in the [try an idea] => [break everything down] => [repeat] stage.
But I already have tokenizing done, and started working on a bit of lexing.
Figured I should comment so that you aren't in the dark.

[Phate6660]

Hitting some breakthroughs! Expect a branch today :D

[TheOnAndOnlyZenomat]

Uh, sounding good. I am exited

[Phate6660]

@TheOnAndOnlyZenomat Got a branch set up: https://github.com/Phate6660/crusty/tree/tok_lex_par

So far with basic testing it should support alpha-numeric characters and quotes out of the box.
Not sure about nested quotes, haven't tried it yet.

[Phate6660]

Obviously it needs more development and testing and all that, but what do you think so far?

[TheOnAndOnlyZenomat]

It definitely looks good. I'll have to take a closer look at the code and play around with the code a bit to fully get it, but looks promising.
Regarding the testing, I will probably sit down sometime and build out our testframework, and will probably try to get some progress done on the prompt.