The New System
Details for the new systemTODO: Move this to Vim roam
This is a file for tracking the new things we’ve done with emulating Roam, incorporating my pre-designed systems from Notion, and mixing the raw managed content in with a Pelican build process for site creation. The following tools allow me to accomplish most of this for now; the details are still being developed. For each section, we demonstrate functionality from each of a few main categories:
- Basic organization capabilities (i.e. wikilinking)
- Task management
- Extensibility/flexibility
- Portability
- Export/import
Ecosystem: Vim
I have a strong prior preference for working in Vim. If I can get everything I need here, I will be completely happy. I’m of course aware that there will be features that are missing compared to other systems, but I am fine with certain tradeoffs.
I’m currently debating the full on use of Vimwiki as a tool for creating file links. For the most part I can take care of everything without it, but it does offer the nice link to Taskwiki. If I decide I don’t need Taskwiki, then perhaps I can go on without it.
Organization
Vimwiki
Vimwiki is the base level Vim plugin that enables the wiki-like ecosystem in Vim. It allows me to easily create wiki pages in a flat hierarchy of Markdown files, and maintains links between those pages as I create them. This is basic wiki functionality, and I’ve done some customization to their defaults like enabling Markdown syntax, using dashes instead of spaces for default file names, and have included additional plugins mentioned below.
The main reason Vimwiki is so powerful is because it enables the easy creation of linked pages that I liked from Roam. All it takes is wrapping a keyword or phrase in double brackets and it becomes page that can store references as well as be a link in a large conceptual web. Although some of the backlinking is not up to par, Vimwiki page creation makes it easy to fly around network of links. There is a small bit of backlinking functionality with :VimwikiBacklinks, but it’s the reference locations and doesn’t juxtapose the content. Perhaps something like this exists, I’m not sure.
Task Management
Task management for the system in Vim can go a number of different ways. Taskwarrior is a fantastic CLI program that can manage tasks all on its own. That said, Vimwiki can integrate with Taskwarrior via the Taskwiki plugin. This offers nice in-house integration so that I can manage tasks by project, and don’t have to leave to update task status. So I have the flexibility to update tasks with Taskwarrior directly outside of Vim, or while I’m viewing the task with Taskwiki.
Taskwiki
Taskwiki is a Vim plugin that complements the Vimwiki task system by making use of TaskWarrior as a backend. By default, Vimwiki has very little support for task managment other than interactive checkboxes:
- [o] Default task
These serve well as an indicator of progress on arbitrary tasks, but that’s about it. There’s no extensive logging/tracking, project assignment, managing due dates, adding priority, etc. Taskwiki adds all this functionality by serving as the middle layer between Vimwiki and Taskwarrior, so you can see your taskd tasks and their metadata inside of your Vimwiki pages. Of course it’s not necessary that you manage tasks inside of Vimwiki (TaskWarrior is fantastic on its own), but it can be very useful to have pages where you can see certain tasks, or simply manage the organization of your tasks visually in Vim. Vimwiki enables you to filter tasks, tag tasks, set task metadata, view reports, etc all within Vim itself. And because it syncs two ways with you TaskWarrior installation, you can still use the handy TW command line to add and reorganize tasks, and later see the changes when going back into Vimwiki.
Taskwarrior
Taskwarrior is a fantastic CLI tool that helps manage tasks effectively from the command line. It makes it easy to add tasks, tag them, set due dates, assign them to larger projects, and see reports. I’m hoping the task management system in Notion can be replaced with the functionality of this program, along with the integration with Vim through Taskwiki.
Extensibility
Vim offers its scripting language and easy ability to customize/remap certain commands. This allows me to (recently) build the file searching and backlinking functionality with relative ease.
Portability
My current Vim configuration is quite portable and well documented. Given the accessibility to Vim on virtually every machine on the planet, and a compact .vimrc, we can replicate pretty much everything on new machines. In terms of access, we have Nextcloud so I can make local changes and be able to build my site from those changes on my web server. With Vimwiki and Taskwarrior, I’m still using Nextcloud to sync relevant files for management across systems. The Vimwiki is just a collection of markdown files current stored under Nextcloud/vimwiki, and .vimrc links Vimwiki to this location as the place to write and create wiki files. The TaskWarrior backend writes by default to ~/.task, but I’ve changed this to Nextcloud/.task in my .taskrc. This allows all the plaintext TW task files to sync up to the cloud, and are available on any system that pulls task from this Nextcloud directory. This is amazingly simple and convenient, and enables me to even modify files from mobile if need be. This kind of setup really eccentuates the idea that the system is not tied down to one tool or machine; it’s a web of mapped ideas available to me anywhere I may need them.
Pelican Integration
Pelican builds a site from raw Markdown files (with proper metadata). Managing the content in Vimwiki would be quite easy; I can have the proper metadata on each page I want the system to build, and that page can be a part of the larger wiki system as well as be picked up by Pelican. The one thing to double check here would file paths; I currently have Pelican rendering files to the wiki or blog or wherever based on their actual file path within content/. We will have to figure out what to do here if we want to maintain a flat file hierarchy with Vimwiki.
PDFs
Viewing PDFs with Zathura is one of the most pleasurable things one can do. It has Vim keybindings, so navigation and search is a breeze. I love it.
Emacs (?)
lolz. Emacs. But actually org-mode is cool, and Emacs as a software package is extremely well developed. Check this for a guide to org-mode should the time come that I really dive into it. Note that this talk is really what even got me interested in thinking about Org-mode as a viable option. I’ve long heard it touted as some magical system that can solve every problem and has more features than can possibly be imagined.
Task Management
From what I can tell, Org-mode has pretty great task management. You can create tasks quite similarly to how Taskwiki manages things, and allows you to filter and add dates quickly for arbitra
Goals
Aside from all the functionality benefits mentioned above, with the new system I want to embrace a few other philosophical and/or less formal ideas that can be beneficial
- Give up structure when it doesn’t help. Ever since I started designed my personal workflows in Notion, I’ve always had a very structural and well thought out approach to everything. But at the end of the day, while things might be nice and organized, I don’t end up actually getting a lot out of the structure because everything’s so neatly filed away. All that information is “put in its place,” but it’s hardly ever seen again, and doesn’t serve a purpose as active, useful information. The truth is a lot of that structure isn’t necessary when you can draw simple connections between your notes. Things become easily indexable via their neighbors, and very quickly you can arrive at useful information you’ve written without having to know and previously placed that information in its “perfect location.” Information in our brains is inherently scattered and interleaved with other concepts and ideas, and Roam’s approach has convinced me that our note systems should be this way as well. So this is a key principle I’d like to embrace. All in all, where you can give up restrictive structure, do so; let go and allow your content to naturally arise from the interconnectivity of the system underneath.
Replacing Notion
I think it’s obvious that a workflow using Vimwiki can do a lot of what I want to do with Notion, especially for managing components dependent on little more than static Markdown files. The question is can it do what I do with tasks and objectives with taskwiki and taskwarrior?
Additionally, being able to maintain basically Notion’s internal Markdown files directly gives me that extra flexibility I need to be able to build HTML straight from my content. It’s the best of both worlds.
Information Management
Knowledge Base
Here I can pretty easily replicate the exact hierarchical structure of the Notion knowledge base with hierarchical wiki pages to serve as directories. Don’t even need to do this though, can just have the endpoint content pages in my flat hierarchy no problem, and reference them anywhere a bit like Roam. This of course adds all the missing features I wanted out of Notion, such as inline maths, I get my Vim keybindings, etc.
Resource Archive
I’ve questioned how I might do this in raw Markdown a little bit before. Seems like a good approach could be to just create a Markdown file with the appropriate meta data I want to track, and take notes on the resource right in that file. This is pretty much exactly what the RA already is, but without some of the nice filtering features. This could perhaps be something available after compiling in the browser i.e. sorting by certain columns or filtering by types. Will have to think about this one a little more.
After thinking about this a little more, and considering how it might be done in Roam itself, I think there may be an even easier way to go about this. Firstly, I think it would be useful have a piece of metadata that indicates whether or not a page is considered an “external source” or part of the “feed”. But its important to remember that these feed items are still pages, just like any other part of the system (and that’s how Roam would treat it). I think in Roam, I would add a resource and its appropriate metadata: URL:
Notebook
The notebook is really nothing more than some Markdown files with aggregate writing and journaling. They are tagged by their type (e.g. daily vs writing) for convenience, which I’m sure could be easily replicated (if needed at all).
Task Management
The main thing is having a proper dating system and managing date sensitive entities with a navigable calendar of some sort. taskwarrior looks promising for this, and can be used directly with Vimwiki via the taskwiki plugin. From what I’ve seen, the task data and management is fantastic, with very enticing graphics for visualizing task progression. You can also assigned tasks to a parent project, which would basically serve as an objective. Events can also be thrown in here, potentially just as normal tasks with an “#event” tag. Maybe we should consider generalizing this so that my Notion tasks and events are really both “task” objects, events being “event tasks” and tasks being “personal tasks”.
Emulating Roam Research
Roam research’s big draw is the interconnectedness of information and the ease with which one can navigate those relations. Here are a couple of its main features, and how they can be emulated here in Vimwiki: - Wrapping a topic in double brackets creates a separate page for that topic. You can then reference that page within other places and begin drawing connections between notes. - This is really nothing more than standard wiki functionality, so obviously the exact same thing can be done here with Vimwiki. The behavior is exactly the same; you press <enter> on a word of visual selection to wrap in double brackets and create a page for that topic in the flat file hierarchy. Roam creates a “references” section at the bottom of each note to show all the places the page has been linked to across the system. That is, it basically copy-pastes the parent block of content making the reference, so whatever was written with respect to that reference is directly viewable (and of course you can click back to the source). This is important for building the connections across concepts, and “back-linking” all over. This is not something Vimwiki appears to do by default, so I need to think harder about how I can get a process to inject these reference sections into the bottom of each note. If it can’t happen live, basically as I’m writing content, it could be part of the build process when turning my content into a nicely styled site. After doing some more digging into this, Vimwiki has a :VWB function to get references to locations around the wiki that reference the current wiki page. This picks out the exact file line and column locations of the references across the wiki with vimwiki#base#backlinks() (which uses internal s:get_links() and resolve_link() methods). It then pushes this information to a “location list” with setloclist() and pops up the window at the bottom. I think it could be quite easy to paste the surrounding context in with these links. Reference base.vim and vimwiki.vim for concrete details. Additional question here is, even if I do get, say, the surrounding paragraph or list item or whatever, what about other wikilinks inside that context? That’s a subtle but nice part about the Roam reference section; it’s not just copy pasted content with context, but it has that same functionality as within the note, so not only the backlinked page is available but those links used inside the context are also available. This is the rich context we’d want here. Another small comment is that, as can be seen with this document, a page can be referenced many many times by the same wiki page. This could result in pretty cluttered previews if not careful. Roam approaches this by having the external file that makes a reference to the current page with a header, and then each of the locations that a reference is page come underneath that primary file header. This is a simple but potentially nice way to clean up the output. Another key feature of Roam is the ability to reference arbitrary note blocks anywhere. It’s as simple as typing “/” to begin searching for a bullet point with a certain phrase in it. Roam will then basically embed that block wherever your typing, you can see the relevant content you’ve already written at the current location. This is a nice feature that I’m not sure I can really do easily in Vimwiki. It would require defining blocks, and being able to properly move content over. Could be quite difficult. Again after some more digging, I’m thinking there may be an angle here. Vimwiki automatically parses “anchors”, which basically allow a wikilink to reference a particle header within the target file. No extra specification has to be done within the target file; a # sign must simply be added before it. If these could somehow be searchable (i.e. with omni complete), then it could be possible to easily link to other block locations and maybe even bring in the surrounding context.
Current Modifications
Fixing Wikilinks
Ultisnaps command to map default style wiki links to markdown ones. There are two issues: - When creating a link, I can type out a word or phrase, go back over it (highlighting it if multiple words), and press <enter> to turn it into a properly titled Markdown wikilink. I can also wrap it in double square brackets and it will automatically create a wikilink, but it will not automatically convert this to Markdown format. When a page has already been created and I’m trying to reference an existing page, I can type it out and hit <enter> again to get that link although this is not preferred with access to autocomplete features. But in order to use autocomplete, your cursor has to be in the parantheses following a normal Markdown link, i.e. []( here. This works fine, but it does not fill out the display text of the link.
So in order to address these problems, we introduce one primary fix: an Ultisnaps command that listens for “[[”. Whenever this is detected, an autocomplete menu is opened, showing all options and filtering as we type. This menu will show the files inside of the wiki that have been created. We can either select from this list, or type a new file name. Once the phrase has been closed with “]]”, it will do one of two things:
- If the referenced link file already exists, simply create the link to that file. The
- default style link will be converted to Markdown by moving the selected file name to
- inside parantheses following the restored spaces text in the preceding square brackets.
- If the referenced link file does not exist, create it, write it, and leave the file. Use
- the typed phrase in the square brackets, and the escaped white space in the parantheses.
This will solve both the creation and reference problems, as well as ensure files are created automatically, even without content. That way, they can later be referenced for their backlinks, despite not holding any actual text.
Better System Search
I recently stumbled across fzf, an incredibly popular and efficient fuzzy search tool. It is can be integrated with Vim easily. We are using it to improve the search experience inside of Vimwiki, primarily due to its speed and the fact it has live fuzzy suggestions as you type. I’m also pairing this with rg to piece together some useful commands that improve the backlink searching and preview experience. There are a few canonical ways of searching in different scenarios:
When searching for lines in the current buffer (the currently opened file), just stick with /. It’s the quickest to type and can match things quickly. Could conisder use of fzf if you don’t know exactly what you’re looking for to begin with and would rather fuzzy find your way through (how to do this is described in the next bullet points). When searching for filenames in the wiki, you can use :Files [path/to/wiki]. This means if I know a specific file name I’d like to open, all I really need to do is get it close and hit enter when fzf finds it. Note that without a specified path, fzf will just recursively search files in the current directory. I’ve currently nmaped <leader>ff (standing for “fuzzy file”) to fuzzy search through file names in the wiki directory (i.e. $HOME/Nextcloud/vimwiki). This is handy because I don’t have to type out the wiki directory everytime I’m not actually there, and my results are limited to those I care about when using the wiki. Note also that I’ve redefined the Files command to include fzf’s default preview window, which is nice touch for seeing the file contents before opening it. Can use :Lines to search all lines of files in - loaded buffers. Probably won’t use this too much since it’s only looking through files I’ve opened in the current Vim session. When searching for all lines across any file, you can use :Rg <query>. This will match all lines with the <query> test exactly, and then open up fzf for a live session with the results (basically a live updating version of :VWS with a preview) Again, when using the wiki we want it’s much nicer to only run search on the wiki files themselves. To do this quickly, I’ve defined the command :Wl <query> (standing for “wiki lines”) to use rg and search only the contents of the Vimwiki directory files. This command also adds the fzf context preview window. Other than these two modifications, the command is exactly like using :Rg directly.
Improved Backlinks (exact and unlinked)
The wiki backlink search interface can be drastically improved by just using fzf for sifting through locations. Here we get virtually exactly what was wanted in terms of interactive search and content preview. There a few added wiki commands that allow efficient backlink, unlinked reference, file, and line matches and navigations:
:Wf: search wiki files usingfzf. Exactly the same as default:Files, but usesfzfpreview and only searches the wiki directory.:Wl: search lines in all wiki- files using
rgon first pattern, thenfzffor live fuzzy search on results. Exactly - the same as default
:Agexcept for added preview and restricting search to the wiki - directory.
:Wb: search backlinks across the wiki for the current file. Usesrgto - match directly used Markdown links to the current file, followed by live
fzfsession - for preview and additional constraint.
:Wu: search wiki for unlinked references to - the current filename. Uses
fzfdirectly for fuzzy searching the filename, and showing - results in live session with preview. (adding to this feature, I should make the matched
- text highlighted when visiting the match so a wiki link can be created by just pressing
<enter>)
Search vs Wikilinks: A Discussion
Honestly this process has me wondering what the point of using Vimwiki and going through the hassle is really for. Finding backlinks is really just search, and with fzf and ag I can very quickly find exact or fuzzy matches for filenames or keywords. Given the power of the commands above, I really don’t need or want to use any of Vimwiki’s search or backlink capabilities. For now, I’m seeing the plugin as an easy way of create links to files; having those explicit wikilinks helps with the web build process. There is also the diary, and the powerful Taskwiki plugin. However, there is a lot that takes away from the plugin, such as poor integration with Markdown overall (wikilinks kinda suck). If there is any easy way to make files and wikilinks without Vimwiki, I really don’t think I need it for aynthing.
I’ve beein thinking about this quite a bit. There are lots of possibilities to play, but at this point, I feel like I should just put it to the test and see what works better.
For getting to files that you want, search is kind of a no-brainer. There’s no tabbing your way down the file, or having to match exactly the name you want. You can just run :Wf to start fuzzy finding a filename using a related keyword. This is crazy quick. That isn’t to say there shouldn’t be listings, because I often don’t remember the thing that I want until I can get a hold of the system or some other files first. It’s just that, in terms of links as navigation, they aren’t very useful since it’s almost always just as quick or quicker to search through the files and open it that way.
I think the main benefit of leaving links in the mix is intent. While the navigational purpose may not be very useful, making a peice of text a link is a pretty undisputable way to show that you wanted the surrounding context to show up when the link was visited (i.e. backlinks), or that you want to remember to think about that phrase as a page with more content. Doing this allows the backlink system to identify all hard matches to a certain filename, so there’s no question about fuzzy matching a vague term or a subset of the keyword or an unrelated context.
Another vote for the wiki: autoformatted tables. These things are really nice I must say, and are a lot like org mode tables. The plugin could be worth it for this reason alone. Note also that these tables are Markdown combatible, so I can do away with the ultisnips I was relying on here.
The Actual System
The new system is Vim, Vimwiki, Taskwiki, Taskwarrior, Timewarrior, fuzzy search (fzf), exact regex search (ripgrep), preview coloring (bat), and external tools (such as Zathura for PDFs), all packaged up together with a group of custom search and linking commands in Vimscript.
Differentiating between single level folders
Right now files are broken into four basic categories: diary, feed, wiki, zettels. I’d like to say the feed and wiki represent the slightly more formal parts of the system, and generally speaking are much more intentional notes. Files in the feed are created entirely for a specific external resource, and most notes inside are derived directly from that source. The wiki includes a variety of personal files along with more “knowledge base” files. Adding knowledge to a KB file or jotting down something about a particular place or person is generally a very intentional task, and is how information is added to the files in these folders directly. On the other hand, the diary and zettel folders constitute a much more informal “on the spot” type of note. The dairy contains daily notes, which can be anything from things I’ve done to arbitrary thoughts to whatever else is on my mind. The Zettelkasten is also a place of informal, self contained thoughts linked amongst each other. This is more of the raw research section, where thoughts are allowed to roam free but are considered something of an atomic unit. Note there is a fine line (although difficult to nail down) between thoughts recorded in a daily note and those included in a Zettel file.
A very important point to consider is how content is added to files across “systems” via the use of (back)links. In Roam research, this appears to be considered pretty standard. A “backlinks” section on each page includes blocks of content linked to the current page from any other place in the system. So it’s not content that was added explicitly to the page itself, but is nevertheless likely very related if not deserving of being on the page. This has been a point of slight confusion for me, because I don’t really like the idea of adding content to a page from another place. For example, people say that they pretty much manage all their content through daily pages, just tagging the target pages with whatever blocks they type out. While I like the idea of making it easy to add content to any page through a single entry point, that information still feels segregated from the main content actually included in that file. This isn’t to say I don’t like the idea of backlinks; I do, and think they’re very important for connecting relevant pieces of information that are tangentially related to each other while being rooted in their own primary topic files. That is, I like the idea of backlinks for linked related content across files, not adding to the file itself.
The Learning Pipeline: Revised
- I think the complete cycle of “knowing” something includes: learning the concept and TAKING FORMAL NOTES, getting hands on with an IMPLEMENTATION IN CODE, and finally continual use of or interaction with the content via REPETITION
- This matches my original system learning pipeline pretty well. Except there was less emphasis on the repetition component, more on learn, build, publish. But to actively maintain that, you need SR in some way, question and refresh your understanding from time to time
- I have the first two: vim system and website render, problib implementation and tool library. Now I just need a repetition system, could be just pull randomly from system? Maybe extract questions? Idk maybe it’s just standard SRS that I need. Someway to keep on top of old concepts