↹lature

2024-03-26    9 minute read
Repository: https://git.sr.ht/~jaxter184/tlature

From the README:

↹lature is a tracker-style DAW. Its most unique features include:

  • First-class support for Open Sound Control
  • CLAP plugin support
  • Terminal UI
  • Keyboard-focused workflow with Kakoune/vim style keybindings
  • Headless mode for portable playback on a SBC or dedicated live performance machine
  • A directly editable, git-friendly file format
An optional aside regarding Bitwig's modular environment Around 2017, I was reverse-engineering the Bitwig Studio internal device files to try and create my own version of the modular environment that they had been teasing since before the 1.0 release. Screenshots of it are hard to find, but there's one at the bottom of [this article](https://sevish.com/2016/microtonal-music-in-bitwig-studio).

Bitwig stores its device files in /opt/bitwig-studio/Library/devices on Linux, and in some old macOS versions, they stored them in JSON format. Though the binary format is (as far as I know) completely bespoke, it is vaguely reminiscent of BSON.

One of the other interesting things that they were doing before 3.0 was embedding their own (JIT?) compiled language called Nitro. As far as I know, there exists no documentation for it online, and I am hesitant to post any example code since basically all existing Nitro code is copyrighted by Bitwig, but it basically looks like C with some special plumbing keywords.

Anyway, I had at some point reverse-engineered it to the point that I could create my own devices with their own UIs! The import process was a bit hacky, and there was a tedious process of dragging and dropping every time I opened Bitwig to ensure that the devices were properly loaded when opening existing files, but I had achieved my goal of having a modular environment where I could make Bitwig devices (though UIs had to be written manually in JSON). My first device was a "notepad" device that was basically just a text box for writing notes in a track.

Fast forward a few months, and Bitwig 3.0 is released, along with some very cool new devices, including "The Grid". The Grid was a more abstract, "friendly" interface that I assume is meant to act as a substitute for access to the real modular environment that they use internally to create their native devices, with the target audience leaning more toward musicians than programmers. Notably, it lacked any tools for creating inline UIs or doing frequency-domain processing (though I have heard rumors that these features are coming at some point in the near future).

But worse (for me), the devices files in the new version were all encrypted (I think). This meant that any new devices released from then on were completely inaccessible to me. While this wasn't too big of a deal since I wanted to make my own devices anyway, this reminded me that this sort of introspection into the inner workings of Bitwig Studio isn't necessarily something Bitwig Gmbh the corporation appreciates. The company's goal is, by definition, to make money, and one of the main ways they do that is by creating new devices that users need to buy additional updates to access. If there were some pack of homebrew devices, this would make it harder for them to sell updates.

I don't harbor too much ill will towards Bitwig, and there are myriad potential reasons for doing this that aren't in conflict with my goals (and are, in many cases, in sync):

  • They want to prevent their future DSP from being accessible by competitors like Ableton
  • They might have been getting support requests from confused people trying to use my tool without realizing it wasn't an official Bitwig tool (I tried to match the Bitwig color scheme with my device editor tool, which in retrospect, was a poor choice)

The true reason(s) I can only speculate about, but either way, I had learned a lot about programming practices and tools from this adventure. pytwig (the library I made that I assume Bitwig is OK with because they haven't asked me to take it down yet) was my first "real" programming project used by people other than just me, and while there's nothing specific to this project that enabled that (perhaps I could have pursued VST development instead to the same effect), I think it gave me the confidence I needed to execute on a project that better represented my own vision.

Fast forward a few more years: free-audio (backed by Bitwig and u-he) releases the CLAP specification, a C-based API for creating audio processing devices. (That's a surprise tool that can help us later)

Honorary mention (but not technically relevant to ↹lature (yet)): dawproject, a DAW interchange format.

motivation

Features that I want in a DAW:

  • text-based save files
    • better git-friendliness
    • readable/editable in a text editor
  • modal editing
    • first-class keyboard-only workflow
    • ideally in a TUI
  • can play audio on a raspberry pi
  • the features listed at the top of this article
  • screen reader compatible
  • package manager for parts that are non-redistributable due to copyright
  • documentation for music (comments in the file, UI elements inline with the plugin chain where you can write stuff)
  • capable of integrating a voice synthesizer into the software

Reaper comes really close, but I think only technically. I feel like it doesn't fulfill the spirit of text-editability, and the modal editing only comes with a plugins like reaper-keys or reaper-nvim. I think in order for a keyboard-only workflow to be actually usable, you have to really rethink what exactly the needs of a DAW are.

The subtext for all of these features is that fundamentally, I want this DAW to be one that panders to my every whim. No matter how ridiculous the feature, if it's something I want, it should be something this DAW does. If these requirements don't speak to you the way they speak to me, you're probably better off using Bitwig or Reaper (both very good DAWs that I still use!).

general design philosophy

  • UI should be limited in scope to a few of the most heavily used parts of the interface
  • similarly, common actions should be bound to the easiest keybinds, and uncommon actions should only be accessible by command line
  • all actions should be accessible by a "scripting" language
  • actions should be easy to type quickly for the "normal" case, but should be configurable (with optional parameters) for esoteric cases

prior art

FOSS DAWs

TUI music software

  • Orca
  • Cacophony (not technically TUI, but certainly TUI-esque)
  • midiTracker
  • any livecoding tool with a CLI REPL or a plaintext file format + hot reloading

trackers

devlog

Controller support

Controller support is implemented using JACK. I initially used midir, but I couldn't figure out how to synchronize notes within the buffer. While the approach that minimizes average latency would be for all the notes to trigger at the beginning of the audio frame, I have heard that psychoacoustically, stable jitter is more important to playability than latency. Plus, at large buffer sizes, having all the messages in a frame trigger at the beginning would likely be noticably distracting.

To implement this, I first add an additional port to the JackProcess struct:

pub struct JackProcess {
	// ...
	in_port_midi: jack::Port<jack::MidiIn>,
}

Then search for a controller and connect it to the port:

for ea_port in active_client.as_client().ports(None, Some("8 bit raw midi"), jack::PortFlags::IS_OUTPUT) {
	if ea_port.contains("HXCSTR") {
		log::debug!("found port {ea_port:?}");
		active_client.as_client().connect_ports_by_name(&ea_port, &format!("{}:midi_in", active_client.as_client().name()))
			.expect(&format!(r#"Could not connect ↹lature midi input to "{:?}"."#, ea_port));
	}
}

Lastly, read the input port in the process loop, adapting the push_event function that I use for auditioning cells:

for jack::RawMidi { time, bytes } in self.in_port_midi.iter(ps) {
	if bytes.len() == 3 {
		let msg = OscMessage {
			addr: "/HXCSTR".to_string(),
			args: vec![OscType::Midi(OscMidiMessage {
				port: 1,
				status: bytes[0],
				data1: bytes[1],
				data2: bytes[2],
			})],
		};
		pd.push_event_root((time as u64, msg));
	}
}

Note that the controller search and OSC message are hardcoded with the string "HXCSTR". Ideally, I'd have both a configurable method to automatically connect controllers as well as a manual menu system in the TUI to set up such connections.

I added controller support so I could practice playing HXCSTR, which unfortunately only supports MIDI at this point in time, but OSC support would likely be similarly trivial.

Chain view

Hypothetically, the entire audio graph can be represented as a list of nodes and the connections between them, where you can see each sound processor as a little "block", and their ports are routed explicitly into each other. However, most DAWs instead have a channel-based workflow, where sound processors are on one of many channels, and the audio from one flows into the next in the series. One of the driving forces that I think about when designing interfaces is to try to make the common use cases easy and clear, and relegate the uncommon use cases to more laborious and obscure parts of the interface. In the case of the audio graph, it is very common for signal chains to be considered on a per-instrument basis, where one set of notes is piped into an instrument processor followed by effect processors. This maps very neatly to the channel-based topology, and as such, I chose to try to preserve that in ↹lature.

The solution I came up with was the "chain", where a Block is a container for a series of Processors. I was struggling with a good way to draw the chain in a way that was scrollable, and this is one of the parts where making ↹lature TUI-based caused some friction in the implementation.

I mostly copied the default behavior of panning in [niri], but for those not familiar, in broad strokes, the requirements were these:

  • The selected processor should always be fully visible (if possible)
  • The chain view should not pan if it does not need to (including when blocks are deleted or resized)

I faffed about a bit for a working solution, but ended up with this:

let frame_width = f.size().width as i32;
// clamp cursor to chain length
self.cursor = self.cursor.min(constraints.len() - 1);
// get the x offset and width of currently selected processor
let cursor_x_offset = constraints.iter().take(self.cursor).sum::<u16>() as i32;
let cursor_width = constraints.get(self.cursor)
	.expect("bounds check should clamp cursor to a valid index of constraints");
// this calculation determines the view offset
self.offset = self.offset.clamp(cursor_x_offset + *cursor_width as i32 - frame_width, cursor_x_offset);

let mut idx_offset = 0;
let mut x_offset = 0;
// because it can be partially cut off, we must calculate the width of the leftmost visible processor
let first_width = loop {
	match constraints.get(idx_offset) {
		Some(width) => {
			x_offset += *width as i32;
			if x_offset > self.offset {
				break (x_offset - self.offset) as u16;
			}
			idx_offset += 1;
		}
		None => unreachable!("offset should not go past end of chain")
	}
};

CLAP TUI extension

source code

TODO: elaborate