Hypermedia browser
Build a hypermedia browser that to begin with works on the file system, using the file:// protocol. These URI/IRIs can be replaced with other protocols like http:// or ipfs:// when we’re ready for networking. It should be agnostic to the things it can browse.
The main view of the browser is a graph, and you can add resources to the graph one by one as things or as a subgraph – a collection of things/nodes. These could be local folders of documents or a remote resource like an S3 compatible/WebDAV interface, or better yet an ipfs or solid (isn’t it just http/WebDAV with a few extras?) resource.
Maybe everything is just a
thingor anode. And if another node chooses to link to that thing with abelongsTo/partOf/graphpredicate, thatthingbecomes at once a node and a graph.
The internal state running on automerge keeps all graphs as separate documents. Allowing for future ease of sharing and contained networking, ie some graphs can be connected/p2p where others can be private (indicated by a lock 🔐 icon or other useful concept).
In the view we need to be able to show nodes and subgraphs at different levels of zoom.
To begin with the application will focus on common image formats and markdown/plaintext.
It allows you to make connections between documents and images, which are kept in the metadata document for each graph. The document is essentially a set of nodes/things that have other resources connected to it.
A markdown file could be just a simple title, like Map research and i can connect all my map images to it. If I want, I can easily extend the text to whatever I can type as markdown.
I can also attach rich texts to the individual images if I wanted. With for instance a caption, credits, whatever. All in their respective files. Combining these with some metadata means that I can easily build complex views of the images and do queries against them.
Question: do all text live in individual files or are they inside the document state? If they exist as automerge documents they could support collaborative editing. (there are ways we could support both, using a file / link icon or similar for the linked file and some sort of animated history icon for the automerge, think discord).
UX HOT TIP: The first time a user launches the app, they will see an example graph of Map research and can start dragging things around and linking stuff at will. Maybe the different icons have animations that draw your attention, and tooltips on hover. The tooltips can be rich at first, and you can choose to only show the summary/title or the whole tooltip. In the settings you can also disable all tooltips for a leaner/more pro interface.
The UI should be customizable with web technologies, firstly with CSS.
Maybe graphs could have associated CSS to make certain nodes stand out or the background different for that particular graph.
The UI should start by introducing the different concepts of the app, the graph and the node (thing/object) and the link (edge)
GRAPHICS: when you add an image resource (MVP supports all web image formats, at least webp, jpeg, gif, png) the app pushes compression and resizing events into a queue that a compressor deamon is listening to. These get made on the device (for now) and stored in an app cache (or alongside the graph automerge document). It’s probably a good idea to store several copies at varying resolutions. Is there a streaming image format out there that supports progressive loading like JPEG? But more efficient and better looking? Is it possible to control which of the progression you store in memory to display on the screen? I also think that as many copies as is manageable should be stored in memory, and have a decay of some sort to make the least used copies of images clear out. Maybe they’re left over from the previous graph we looked at? We don’t need them. Maybe using a browser wrapper like Tauri or electron would deal with this kind of garbage collection for us.
MEMORY: is is essential that this app is resource efficient. It should use as little memory as possible, and bring up compressed images from a local directory, not load the originals until they are requested. At a given zoom level we can give a button with an icon to bring in full resolution/original file instead of preview.
Settings for types of preview should be possible to adjust manually after a while. Maybe you’re on a low resource device and you only want to build previews of a certain size?
GRAPHICS: is it possible to compress images on the GPU? The new M1 Macs should be very good at running the compression algos in the background.
By the way, I think that the layout of the graph should be encoded into it (in its own layout object, btw we should support multiple custom layouts), with the option to turn on auto/force layout (and in the future, multiple views based on machine learning). This way it’s easy for the user to make the graph a very powerful tool for laying out ideas in space, grouping things visually and manually. This is very important (and in the MVP, more important than auto layout)
Resources
Rust and graphics image manipulation.
Could also use bindings to C/C++ binaries like libvips or imagemagickbut I don’t know if they support GPU acceleration.
Photon. High-performance, cross-platform Rust/WebAssembly image processing library. https://github.com/silvia-odwyer/photon
Piet. Cross-platform 2D graphics. httpshttps://github.com/linebender/piet/blob/master/README.md
High-level Interface to NVIDIA® CUDA™ Driver API in Rust https://github.com/bheisler/RustaCUDA
Tutorials to get me started
Practice Rust and TAURI: Make an Image Viewer https://medium.com/@marm.nakamura/practice-rust-and-tauri-make-an-image-viewer-1-c8e08427340d and #2
Version 0.0.1
First of all, just build a simple app where you can drop files into and display them on a canvas. Log out all the image file paths.
Version 0.0.2
Second step is to add these to a graph with layout and store the state to a persistent storage on disk. You should be able to reopen the app and get the same state.
Version 0.0.3
Each graph can be opened as its own tab or window, or like a web browser, you can just click into one to replace the current view. The app state with which graphs are open, the zoom level and position of each tab in relation to its view is stored in a separate app state.
Continue writing this version plan for the app development. Tiny incremental and meaningful changes for each patch version. Will be like a to-do and bring meaning and drive to versioning.
Note from 2022-02-23
Not that I thought this, or any of my related ideas were unique, but it’s interesting that these ideas keep popping up, being thought about in much the same way for many, many years. Check out the Object browser by BILL SEITZ. His Wiki has some super interesting links.