Ark

ark is my latest personal service, after my own rss reader. The stack is the same as the other services, and it all lives in the same repo.

ark stores markdown documents, and that's basically it. It also uses qdrant for embeddings for the docs so that I can do similarity search. The embeddings come from OpenAI's text-embedding-3-small model, which have cost me $0.01 so far.

This is what it looks like; colors are subject to change:

I'm considering changing the separate-edit page with a heavier inline markdown editor, but it might be easier to let the cli take care of editing:

$ ark
Usage: ark <COMMAND>

Commands:
  add      Create a new document [aliases: a]
  get      Get a document by ID [aliases: g]
  update   Update a document by ID [aliases: u]
  delete   Delete a document by ID [aliases: d]
  search   Search documents by meaning (or exact text with --text) [aliases: s]
  browse   Browse recent documents interactively [aliases: b]
  list     List recent documents [aliases: l]
  reindex  Reindex all documents for search
  help     Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

I inserted my entire Obsidian knowledge base into ark and it is about to replace Obsidian for me, although it should be said that my usage has been very low for years. Searching uses the embedding model for similarity search and outputs the ids and scores for the top candidates:

$ ark search 'integrate 0-form over manifold'
[743] (0.64) ddg/Exterior Calculus.md
[594] (0.53) oaomm/manifold.md
[418] (0.51) ddg/Ex/Ex6-3.md
[631] (0.50) oaomm/immersed submanifold.md
[555] (0.48) oaomm/sphere manifold.md

ark only tracks markdown data, it doesn't track files. These results look like filenames, but this is a quirk of the Obsidian import: I had previously used the filenames of the obsidian files as relevant metadata so that the content itself wouldn't necessarily make sense on it's own. To keep it, I prepended the filename as the first line of every file.

You can then get the markdown from an id:

$ ark get 743
ddg/Exterior Calculus.md

$\def\RR{\mathbb{R}}$Let $f,g$ be 0-forms and $M$ some $n$-dimensional manifold. For each point $p \in M$, the form $f$ assigns a number $f_p\in\RR^n$.

We can integrate a $0$-form over the manifold, like so: $\int_M f$. By this we mean the sum of all $f_p$ where $p\in M$.

### Differential
The differential operator $d$ takes a $k$-form to a $(k+1)$-form.

ark browse basically does these two things in one interactive step.

The beauty of ark is that this is it. It's that simple. But simplicity means that it's easy to integrate with. Here's a claude skill for interacting with ark:

---
name: ark
description: CLI tool to handle personal markdown documents.
---

`ark --help` shows the help page.  Responses are in json.
`ark search TERM` uses vector embedded similarity search.
`ark search --text TERM` uses exact search.

Now I can ask claude for how to make pizza the way I want it:

$ claude 'use /ark to find what I need for an italian dinner' --allowed-tools="Bash(ark:*)" -p
Here's what I found for your Italian dinner:

**Pizza Recipe** (for two pizzas):

Dough:
- 200g water
- 7g salt
- 1.0g dry yeast
- 275g flour

Sauce:
- Canned Italian tomatoes (from IMS)
- Dry oregano
- Squeeze and cut large pieces, don't put on too much, spread it out to the crust

**Wine pairing:**
- Wongraven Alleanza Langhe Rosso 2024 (your favorite!)

This is a silly example because the llm didn't do anything useful in this case, but it's using my data to help me. The data comes from two notes, one for a pizza recipe and one for that wine, which I put in as a test. This is my pizza recipe, but the wine isn't my favorite.

What I like the most about ark is that I have access to my entire set of notes from anywhere, and can trivially get other tools to read it and do useful things with it; my data isn't held hostage by a SaaS. It's also very much unstructured, and I hope to be able to keep it that way. Leaning on other tools for search feels more appropriate for my use case than having to structure files in a tree or manually annotate with tags.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License