LAB NOTES
Some ideas need a place to run.
We write up our experiments and share the open-source tools we build.
Research &
experiments
Notes on what we are building, how it works, and what we want to test next.
Open-source work
Tools and libraries we maintain ourselves.
- RepositoryNicheBenchOur Python CLI for evaluating LLMs on framework-specific tasks, starting with Drupal. The public repository contains the harness and documentation; task data needs separate access.
- Quiz, code-generation, and bug-fixing tasks with a model under test and a judge.
- Opt-in isolated Drupal/DDEV runtime checks, collected artifacts, and a final diff.
- Task-specific scoring rubrics, configurable weights, and critical-check gates.
- RepositoryWhisper Recorder UIOur browser recording PWA sends audio to a configured speech-recognition endpoint. You choose a reachable compatible server and model, such as Whisper or Qwen3-ASR; models run on the server.
- Record audio in the browser and send it to a local or OpenAI-compatible ASR endpoint.
- Add hints for technical vocabulary. You could configure LangChain or Langfuse hints when those terms are misheard.
- Optionally process the transcript with an LLM, keeping raw and processed text separate. Installable on supported browsers; transcription requires a reachable server.