The Magic Whiteboard - Part II
Subtitle: The AI Librarian
# About the Librarian
In the previous post The Magic Whiteboard - The Obsidian Wizard we showed you how to set up the whiteboard using the note-taking application Obsidian. Now, we’ll extend the project by creating the AI Librarian. Most of the AI Librarian will be constructed from links in The Academic Browser with a few new additions.
The idea for The Magic Whiteboard was partly inspired by Curvenote “A collaborative writing and publishing tool for scientists & researchers” although I’d been thinking about something similar for a while.
The goals of The Magic Whiteboard are to have a self-contained AI-assisted workspace to help through the entire process from the initial concept to the final published article. Curvenote provides a workspace for writing a document and can export to many standard formats, links to JupyterLab notebooks, and provides real-time collaboration. The Magic Whiteboard also has all of these features, but The Librarian puts AI-based literature search at your fingertips.
Curvenote has a free option, but charges for journal templates, more than one private project, and extended online storage space. The Magic Whiteboard is built from free components and the only fees are for the more capable AI chat models.
All of your documents and files are stored locally, unlike the online service Curvenote, but this means that you’ll have to install Obsidian, plugins, and any other software you want to use. The advantage is that you have full control over your content.
# The Librarian’s Links
Literature searching is evolving very rapidly, especially with the inclusion of Large Language Models (LLMs) and the ability to query in natural language. One of the best resources is Aaron Tay’s Musings about Librarianship, and his articles Using Large Language Models to generate and extract direct answers and List of academic search engines that use Large Language Models for generative answers.
These are some of my favorites, but you might prefer others.
# Search and Encyclopedias
These give broad searches to get started.
# Literature Search
To find journal articles on a topic, use these dedicated search engines.
- Inciteful - Find the most relevant literature, faster.
- RefSeek - A web search engine for students and researchers that aims to make academic information easily accessible to everyone.
- OpenAlex - A free index for the world’s research.
- Scienceopen - A discovery platform with interactive features for scholars to enhance their research in the open, make an impact, and receive credit for it.
- BASE - One of the world’s most voluminous search engines especially for academic web resources.
- CORE - The world’s largest collection of open-access research papers.
- arXiv - Free distribution service and an open-access archive for nearly 2.4 million scholarly articles in the fields of physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics.
- PLOS - Open Access publisher empowering researchers to accelerate progress in science and medicine by leading a transformation in research communication.
# AI Search
AI-aided search can help narrow the focus of your literature search.
- Consensus - Uses AI to find answers in research papers. The best way to search is to ask a question.
- SciSpace - Do hours worth of reading in minutes. SciSpace is an AI-powered tool to help you understand research papers better.
- Scinapse - AI-powered Research Intelligence Engine
- Perplexity - AI-chat-based conversational search engine that delivers answers to questions using language models.
- Semantic Scholar - A free, AI-powered research tool for scientific literature.
- Dimensions AI - Linked research data from idea to impact.
- R Discovery - Accelerates your research discovery journey, with latest and relevant content in your area of interest.
# Graphical Search
Graphical searching makes it easy to see other relevant papers.
- Connected Papers - Explore connected papers in a visual graph.
- Open Knowledge Maps - A visual interface to the world’s scientific knowledge.
- Citation Tree - Grasp a fundamental understanding of a field from a couple of core papers.
- Citation Gecko - Helps you find the most relevant papers to your research and gives you a more complete sense of the research landscape.
- Litmaps - Discover the world of scientific literature
- Concept Map - Provides a more visual and interactive approach to exploring topics with artificial intelligence.
# AI document analysis
Try some of these to get a deeper understanding of articles. These are free to use but may have limits on the number of documents you can upload per month, so when you hit a limit on one, try switching to another analyzer.
- ChatPDF - Answers questions about a research paper.
- Bearly - Intelligently parses and understands information in uploaded papers, and provides tailored answers to your queries.
- docAnalyzer - Dynamic, context-aware interactions with one or multiple PDFs, powered by cutting-edge AI research for superior document analysis.
- PDF AI - Upload a PDF and have a chat with it.
- OpenRead - OpenRead PDF parser leverages AI technology to parse PDF papers into interactive papers. You can learn the AI summarized paper from Paper Espresso, ask paper questions from the Paper Q&A, and take notes on digitized paper and turn them into your personal learning library.
- Docalysis - Save time by having an AI answer questions about your files within seconds.
- Sharly - Employs generative AI to help professionals grasp lengthy papers faster. It uses the most recent language models and natural language processing to streamline long, complicated papers.
# AI Chat
Ask questions and receive answers in natural language. Most of these require registration, but there is a free tier. Anthropic will let you upload a document for analysis, and all of them can be used in the writing process. We used Google Bard to write an outline in Part I of this series. The Obsidian plugin Surfing will open any of them in a new tab. You may also want Copilot and/or Smart Connections if you’d prefer to remain on the page.
# AI Authoring
These tools will help you write or revise your document using AI.
- textero.ai - Use the power of AI to overcome writer’s block, find new ideas, and elevate your writing experience with Textero.ai.
- Minerva - The world’s most advanced virtual research assistant for academics, researchers, and students.
- Isaac - Think of it as your own ChatGPT specifically designed for academic writing.
- Outline - Outline is an AI-powered collaborative document editor.
- SpinBot - Intelligent, free text rewriting tool.
- Trinka - An online grammar checker and language correction AI tool for academic and technical writing.
- AcademicGPT - Writes abstract, conclusion, summary, and future work sections from uploaded document.
- Expresso - A tool to edit texts and improve your writing style. It will teach you to express yourself through writing more efficiently and help make your texts more readable, precise, and engaging.
# Online LaTeX Editor
Many journals require submissions to be in LaTeX format. You can install TeXStudio or LyX locally along with LaTeX compilers such as TeX Live or MiKTeX. Another option is to use Overleaf which is a collaborative writing and publishing tool and LaTeX editor.
Overleaf has templates for most journals making the writing process much easier. They charge $199 for online collaboration of up to 10 authors, but you can work alone for free. An open-source alternative is SwiftLaTex.
Bit.ai Built for teams and individuals to create, collaborate, and organize all your knowledge in one place from anywhere in the world. Create fast dynamic notes, documents, wikis, knowledge bases, projects, client deliverables, technical docs, training guides, and client portals, while integrating across the apps you work with. The free plan has enough features that you should be able to collaborate with your colleagues.
# Reference Manager
Zotero is a free, easy-to-use tool to help you collect, organize, annotate, cite, and share research. Several plugins help integrate Zotero with Obsidian. Alexandra Phelan explains in “An Updated Academic Workflow: Zotero & Obsidian” the steps she uses to connect the two apps. You should install betterbibtex for Zotero (Tools Add-ons). She recommends ZotFile but it is no longer needed in version 6.0 and later for Zotero.
Plugins for Obsidian are Zotero Integration, Obsidian-Pandoc, and the Pandoc Reference List. Pandoc converts documents from one format into another, so you can write in Markdown in Obsidian and then convert to LaTeX, HTML, or MS Word to publish. You might also want to include a link to Citation Style Language (CSL) for ZoteroBib to format references in a style compatible with your chosen publication.
Phelan also recommends the Dataview and Admonitions plugins for Obsidian. Dataview allows you to query your Obsidian vault and to search and display data as well as create summaries of referenced literature. Admonitions " is a tool that allows you to create attention-grabbing callouts, tips, warnings and other informative blocks within your notes".
Under “Community Plugins” in the Settings window of Obsidian, click on “Zotero Integration”. At the top, just under “General Settings”, install the PDF Utility, and then under “Citation Formats” choose “Template” from the “Output Format” drop-down list.
Copy and paste the template from the article under section III. Literature Notes. Under “Bibliography Style” you can select a publisher style by typing in the box. For example, if you type iee the list will expand to show IEEE, IEEE (with URL), and IEEE Transactions on Medical Imaging as the first three choices.
In “An Academic Workflow: Zotero & Obsidian” Phelan describes how she uses different highlighting colors to annotate a PDF:
Annotation Color Scheme
Ideally, the reference system would be fully bidirectional. That is, you should be able to insert a reference into your document and just as easily insert a new reference into Zotero. With the Zotero Integration plugin, creating a reference in the document is very easy, and going the other way is almost as simple.
Choose one of the methods provided by The Librarian and begin your literature search. I selected Perplexity to ask about optimally fitting Bezier polynomials to curve points. Perplexity provided a complete answer as well as a few relevant papers. One of the papers is in arXiv and after copying and pasting the DOI (Document Object Identifier) into Zotero, Zotero finds the bibliographic information online and imports it into the current folder.
Zotero Bibliography Lookup
# The Zotero-Obsidian Link
To be able to link Zotero and Obsidian, add the Shell Commands plugin to Obsidian. From the plugin documentation, make a new command by clicking on “Shell Commands” in the left column. This opens a new window,
The Shell Commands Plugin
where you can insert a link to the shortcut that starts Zotero.
Test the link by clicking. It will take a few seconds for Zotero to start, but your library of references should appear. To add a new entry, see Adding Items to Zotero and scroll down to “Add Item by Identifier”. By clicking on the wand in the top bar, you will be able to enter the reference ID, and Zotero will automatically find the reference and fill in the details.
# The Librarian Layout
The Librarian relies on external websites that open in new tabs in Obsidian. To help keep your links organized, you might want to install the Multi-Column Markdown plugin. To create a multiple-column region, begin by typing three backticks, followed by start-multi-column. Enter a unique ID such as The_Librarian and enter the number of columns.
Enter the first column of links just below the second row of backticks. Headers are generated with hash marks (#) and bullet points with
Ctrl-]. Put the source name in square brackets and the URL in parentheses immediately after to make hyperlinks
[Google](https://www.google.com/). You can also highlight the name and then type
Ctrl-k to automatically insert the brackets.
The link for Zotero is made using the Shell Commands plugin described above.
At the end of the first two columns insert and after the end of the third column . Click out of the multi-column region to see the results:
The Librarian Tab
The last component we need for The Magic Whiteboard is The Hacker which will use AI to help you write code for your STEM projects.
# Image credits
- Hero: Dream Studio “A robot librarian in the stacks of a library, reading an e-book.”
- Curvenote Video: Curvenote collaborative writing and publishing tool
- Annotation Color Scheme: Alexandra Phelan, “An Academic Workflow: Zotero & Obsidian”
- Zotero Bibliography Lookup: Zotero
- The Shell Commands Plugin: Shell Commands
# Code for this article
- Obsidian and plugins
A free, easy-to-use tool to help you collect, organize, annotate, cite, and share research.
If you need to convert files from one markup format into another, pandoc is your swiss-army knife. Pandoc understands a number of useful markdown syntax extensions, including document metadata (title, author, date); footnotes; tables; definition lists; superscript and subscript; strikeout; enhanced ordered lists (start number and numbering style are significant); running example lists; delimited code blocks with syntax highlighting; smart quotes, dashes, and ellipses; markdown inside HTML blocks; and inline LaTeX.
Posts using Lyx
See all software used on wildpeaches →