Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature request] BibTeX / Literature in Franklin #555

Open
kellertuer opened this issue Jul 16, 2020 · 20 comments
Open

[Feature request] BibTeX / Literature in Franklin #555

kellertuer opened this issue Jul 16, 2020 · 20 comments

Comments

@kellertuer
Copy link
Contributor

Thanks for this nice package, it looks neat and seems easy to use so far.

Would it be possible to integrate for example Bibliography.jl to have a Publication List (maybe even literature in single pages/blog posts at the bottom)? That would be awesome. Especially a publication list would be nice to have (and my starting point to not only play with Franklin but actually switch).

@tlienart
Copy link
Owner

So Bibliography.jl doesn't work for me (it's also not registered) but here's a prototype with BibTeX.jl which should show you how things can work; this is with the sandbox template

index.md

# Test with BibTex.jl

Some text here etc

## References

{{show_refs coleridge almendro maron}}

utils.jl

using BibTeX

function ref_item(ref, infos)
    io = IOBuffer()

    author = infos["author"]
    author_last, author_first = strip.(split(author, ","))

    write(io, "<li id=\"#$ref\">")
    write(io, """$author_first $author_last, <span style="font-style:italic;">$(infos["title"])</span>, $(infos["date"]).""")
    write(io, "</li>")
    return String(take!(io))
end


function hfun_show_refs(refs)
    _, allrefs = parse_bibtex(read(joinpath("_assets", "bibex.bib"), String))
    out = IOBuffer()
    write(out, "<ul>")
    for ref in refs
        infos = get(allrefs, ref, nothing)
        isnothing(infos) && continue
        write(out, ref_item(ref, infos))
    end
    write(out, "</ul>")
    return String(take!(out))
end

and that's it. The example bib file I'm using is this one and is saved in _assets/bibex.bib

Result

Screenshot 2020-07-16 at 12 37 01

Notes

  • If you decide to use what I'm showing above, you'll have to add a bit of logic to remove these {...} and parse things like accents which are stored in the LaTeX format (see screenshot); this is pretty easy with a few replace in the ref_item function; though ideally these are things which should be done in the packages themselves...
  • Franklin has an internal way to handle references (parsing and processing of \cite, \citet, \citep); it could be hooked up with something that can parse bib files without too much effort but I'm currently a bit disappointed by the state of these libs (though you may have had a better experience in which case a working example using one of those packages would be useful)

@tlienart
Copy link
Owner

tlienart commented Jul 16, 2020

hmm this could do the job: https://lucianolorenti.github.io/BibTeXFormat.jl/latest/ though it can't handle accents apparently; a good start nonetheless.
accents: lucianolorenti/BibTeXFormat.jl#6

@tlienart tlienart added enhancement New feature or request question labels Jul 16, 2020
@kellertuer
Copy link
Contributor Author

Oh sorry, I wasn't aware that Bibliography.jl is not yet registered, I just saw a propotype with that and Documenter recently and since I am thinking of moving my page from Jekyll to Franklin and this is the only thing I am not yet sure about... this might be a good thing to discuss here.

So yes of course, BibTeXFormat and things are a good way, too. I will check.
Concerning Accents, my usual bib-files are from BibLaTeX using UTF8, then accents should also be not a problem.

@tlienart
Copy link
Owner

Right then in your case you should be able to "just" use my solution tweaking for the desired look. You could also use BibTexFormat but I'm not sure it will help you massively

@kellertuer
Copy link
Contributor Author

I will try that then, maybe next week maybe during JuliaCon and report back.

@j-fu
Copy link

j-fu commented Oct 11, 2020

Hi, while the above variant with BibTex.jl essentially worked (kudos for being able to parse my messy bibfile after only a few fixes!), presentation of the results in html is quite hard to achieve. OTOH if you have pandoc and pandoc-citeproc installed, you can make use of the vast library of CSL styles available, e.g. you will find there ieee.csl.

Assuming a bibliography mypapers.bib in _assets, this can be achieved in the following way:
Write a "hfun" calling pandoc which writes a the html formatted bibliography to stdout:

function hfun_pub()
    read(`pandoc -f markdown+yaml_metadata_block+citations+raw_html _assets/pub.md --filter pandoc-citeproc --csl=_assets/ieee.csl --bibliography=_assets/mypapers.bib   --mathjax`,String)
end

In pub.md, you just write the yaml header with the bibtex keys (these dots etc. matter) :

---
nocite: |
 @Article1,
 @Article2,
 @Article3,
...

E.g. in publications.md you write

# Publications
{{pub}}

I had this up and running for some custom static site building, now I am a proud owner of a
Franklin.jl based homepage ;-).

@kellertuer
Copy link
Contributor Author

This looks really really nice, thanks Jürgen! I will take a closer look hopefully soon.

@tlienart
Copy link
Owner

beautiful! that's something I can add in the docs as a nice example

Btw @j-fu are you happy for me to add a link to your webpage in the README? thanks!

@j-fu
Copy link

j-fu commented Oct 12, 2020

Well after sleeping over it:

  • The ultimate julian way would be to implement CSL based typesetting in Julia, which seems to be an achievable goal, given that there are already several implementations in different languages.

  • The pattern to use pandoc could easily be generalized to the processing of whole markdown pages. In particular, this would allow to have in-text references. Not sure though if this should be the way forward here, but pandoc is just incredibly powerful (and thus making a point for haskell btw...)

Albeit interesting, all of this is outside my priorities, so i just mention these points here for anyone liking to pick them up...

As for linking my home page - well yes, if you think it is ok enough - I won't be that active maintaining it, tough I plan to post my scientific computing course material (julia/pluto based) there in the next couple of months.

@kellertuer
Copy link
Contributor Author

I would contribute to both a CSL based typesetting as well as something that does pandoc-ish things with Franklin, though also for me both topics are not my main priorities so I will not start such projects but would contribute if they exist.

@tlienart
Copy link
Owner

tlienart commented Oct 14, 2020

So I had a look and it seems to me that a good working compromise is to just use citeproc-js which is heavily battle tested.

In a way this would be similar to the pandoc option that @j-fu used but the project seems (more?) active and tested, also there would then be two options, one where you just load the js library and one where you pre-render. Franklin could provide some glu code to make this seamless. In a way this is similar to the code highlighting, there could be a Julia based highlighter and in fact there are a couple but they are far from as battle tested as highlight.js or prism.

It might then appear that this is somehow too slow or not flexible enough and might seed an effort towards a Julian one but this is an entire project in itself and doesn't look trivial (many finicky details).

Ps: on the pandoc front, Franklin is now a bit too far, so there could still be a plugin where the user would say "give that bit to pandoc and plug the results here untouched" (potentially an entire page) but pandoc cannot do some stuff that Franklin does and using both would likely be more annoying than helpful :)

@adigitoleo
Copy link

adigitoleo commented Aug 13, 2021

If the references can be processed before building the site, DOI content negotiation could be of interest. So long as all the publications have a DOI that supports it, the plain text reference can be retrieved with e.g. curl -D - -L -H "Accept: text/x-bibliography" <doi link here> (remove -D - to hide the header info) or using a simple script that can do web requests.

@tlienart
Copy link
Owner

hmm pretty cool and definitely doable

@adigitoleo
Copy link

I'll play around with it a bit, for now I'm pasting each entry in manually. My first hunch for automation is that the output might still need some touching-up, like turning the DOI number into an actual link.

@adigitoleo
Copy link

adigitoleo commented Mar 16, 2022

I've come up with something that will probably suffice for me for now: https://gist.github.com/adigitoleo/52d793e5fa4a8d4787ff556fae7f9454

  • Doesn't rely on bib file, you only need a TOML file with some DOI links
  • Citation is retrieved from crossref (or whoever services the content negotiation for your DOI)
  • Formatted responses are cached to avoid repeat requests (delete the cache file to reset things)
  • Authors are put inside <strong>, title in <em>, links are hyperlinked
  • Can refer to the entry by the same key used in the TOML files
  • Can put arbitrary text in the in-text citation (it's just a dumb href link)

It's a bit verbose for multiple citations at the moment:

{{{citeref key1 Foo & Bar (2022)}}, {{citeref key2 Foo et al. (2022)}}

Maybe I'll get around to making it nicer at some point. Also I haven't tested it very much and it is past midnight so please check it before using. Regex might need some tweaking.

@tlienart
Copy link
Owner

Very nice! and I love the big regex in the middle, does the website for which you're using this have a public repo? in which case it might be good to link to it too so users can see a demo and potentially see your latest updates.

@adigitoleo
Copy link

adigitoleo commented Mar 17, 2022

Working on my first blog article now, then I'll push it as an example, with a fix for the regex which didn't work for textbooks (or anything that doesn't have a journal name). The retrieved reference text has so far been fine in most cases, occasionally I get strange things in the crossref metadata (usually something in place of missing/incorrect page number declarations). But it is easy enough to touch up the refcache. I've also made the citation macro print ??? like the default one when a ref is missing from the list...

@adigitoleo
Copy link

adigitoleo commented Mar 24, 2022

All of the stuff can be found in here: https://sr.ht/~adigitoleo/adigitoleo-website/sources Changed to using Zola (it's just so dang fast).

For example, utils.jl.

The article with references: https://adigitoleo.srht.site/blog/geophys/drex-tests-analytical Removed, some info migrated to Python API docs.

Website also linked on my github profile. The article is very technical and mostly for self-reference. Don't worry about the content. It did however serve as a useful test for just about all of the features I will ever need (maths, images, flexboxes, refs, footnotes...) Happy to be added to README here, although that's basically all I have there at the moment.

@lrsantos11
Copy link

Hi, while the above variant with BibTex.jl essentially worked (kudos for being able to parse my messy bibfile after only a few fixes!), presentation of the results in html is quite hard to achieve. OTOH if you have pandoc and pandoc-citeproc installed, you can make use of the vast library of CSL styles available, e.g. you will find there ieee.csl.

Assuming a bibliography mypapers.bib in _assets, this can be achieved in the following way: Write a "hfun" calling pandoc which writes a the html formatted bibliography to stdout:

function hfun_pub()
    read(`pandoc -f markdown+yaml_metadata_block+citations+raw_html _assets/pub.md --filter pandoc-citeproc --csl=_assets/ieee.csl --bibliography=_assets/mypapers.bib   --mathjax`,String)
end

In pub.md, you just write the yaml header with the bibtex keys (these dots etc. matter) :

---
nocite: |
 @Article1,
 @Article2,
 @Article3,
...

E.g. in publications.md you write

# Publications
{{pub}}

I had this up and running for some custom static site building, now I am a proud owner of a Franklin.jl based homepage ;-).

@j-fu I just contructed my publication list using your ideias. Thanks for that.
I have a question: In your webpage the publications are sorted by date. Did you change the ieee.csl class to reflect this? Could you give some hint on how you did it?

@michaelgutmann
Copy link

michaelgutmann commented Sep 24, 2024

Thank you all for the helpful discussion.

I wanted to parse bibfiles with my publications and display them neatly on my homepage, together with links to arXiv or the publisher's sites. The above solutions didn't quite work for me so I resorted to the following solution that may also be useful for other people with this use-case:

  • I first import and sort the bibliography using the Bibliography and DataStructures package (code)
  • I then extract the content that I need from the created datastructure and do some string manipulation for a nice conversion to html. The solution is a bit hacked together and specific to my needs but works and can hopefully be adapted to other people's needs too (code)
  • Note that the Bibliography seems to be able to do chronological sorting (oldest publication on the top) but not reverse chronological sorting (newest publication on the top). Reverse chronological ordering can be achieved with Iterators.reverse when looping through the papers, see here

That's how it looks like: link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants