Task List

• Up Next

• Soon

  • Allow passing additional dependencies which can be used in the scripts
    • This would probably be paths to somewhere on the filesystem that contains a library with package.json and all.
    • Or maybe just a simple package.json with node_modules directory and then the script can be run with that as the CWD
  • Support callbacks from the JS side to the Rust side
    • e.g. to do expensive lookups, mutate state, etc.

• Later/Maybe

  • Consider using worker_threads instead of cluster
    • This should use a bit less memory (though probably not much less), but is complicated by the lack of ability to send sockets across thread boundaries, unlike cluster.
    • The bigger advantage is that it will make it easier to maintain a single code cache, and share data between threads in a zero-copy manner, using SharedArrayBuffers.
    • One approach is to have a single master thread create a new MessagePort pair for each connection. A thread gets the port and then the master shuttles buffers back and forth between them.
      • Mostly implemented here https://www.phind.com/search?cache=zdy35n834rpmds1cmiiszwt8.
      • Feels bad for latency though
    • Another way would be for each thread to have its own socket and then the Rust side communicates with each one and handles the load balancing. This would also potentially make it easier to share other things like persistent contexts but that is probably over complicating it.
      • This isn't great as a general approach but works well for this case because we know that there's just a single client (the Rust process) with exact knowledge of how things are structured.
      • I'll probably go this route.
    • Need to benchmark these approaches
  • Support multiple engines (Bun, Deno)
    • Bun doesn't support node:cluster yet
    • Deno should work with minimal changes
  • Run entire projects
    • with node_modules and so on
  • Support running Boa or QuickJS in-process with the same interface
  • Allow control over which Node built-in libraries can be imported
    • Currently none can be imported, this wouldn't be hard to change but making it configurable would be great.
  • Network sandbox
    • This can be done most easily by replacing fetch with a function that wraps fetch and checks the URL before passing on the request
  • Filesystem sandbox
    • Same as network, but for the fs functions. This is a bit more difficult to get right since there are so many functions.
    • Ideally we could just run the isolate in a chroot, but I don't think that's possible.

• Done

  • Benchmarks
  • Support V8 code cache when evaluating code
    • Need some kind of content-addressable LRU caching so we don't hold on to code cache values forever
  • Customizable number of workers
  • Improve Rust API for an engine connection
    • Single function to run a script and wait for its result
  • Start a single engine
  • JS code to handle requests, run code, and respond
  • Send requests to the engine, get a response
  • Support one-way communication of a script back to the Rust side (e.g. progress messages)
  • Ability to reuse a context across multiple requests without reinitializing it each time.
  • Hook into console so that it goes back to Rust in a way that is linked to each request.
  • Run a pool of engines and load balance requests between them.

Basic Design

• Rust Side

  • Start and maintain a pool of engines
  • Allow passing JS code to the engine, either as strings or as filesystem references
  • Support callbacks which

• JS Side

  • The core is a small application which takes in requests
  • Support auto-importing modules from the filesystem before running the expression.
  • Support compilation snapshots, if possible
    • Looks like Node.js support this via createCachedData. It doesn't cache stored data but does cache functions.
  • Library which can callback to the Rust side to take actions

Alternatives

• Why not Deno?

  • In my experience in the past this worked out ok, but
    • you have to set up a lot of runtime stuff yourself
    • Updating all the Deno crates together is a pain and there were often breaking changes to be handled.
  • Some of this might be better now as Deno itself has matured, but overall it seemed that embedding a "full-featured Deno" is not really as easy as it should be.

• Why not bindings to QuickJS/Boa/etc?

  • Runtime compatibility. Some applications may need to do things that only really work in the most popular JS engines, such as talking to Postgres.
  • Harder to include arbitrary NPM packages or similar, without bundling
  • For applications that don't need this, I do think it's worth providing a mode that just uses bindings and won't have to start a separate sidecar process.
  • As QuickJS gets more WinterCG compatibility this also may be less of an issue.

Introduction

• The paper defines agents as a system that uses planning, loops, reflection, and other control structures, as well as leveraging the model's reasoning abilities to accomplish a task.
• This paper focuses mostly on the difference between single-agent vs. multiple-agent architectures.
• The multiple agent architectures are then subdivided into vertical and horizontal architectures.
• Each agent has a persona which is basically the system prompt as well as the tools that the agent has access to.
  • In addition to the instructions for the task, the persona may define a specific role such as an expert coder or a manager or a reviewer and so on.
• Tools of course are external function calls that the model can request, such as editing a document, searching the web, or other actions that the model is not able to do inside its computations.
• The paper defines a single agent architecture as those powered by a single language model which performs all the tasks on its own with no feedback from other models or agents. There may be feedback from humans though.
• In a multi-agent setup, each agent typically has a different persona.
• In a vertical multi-agent architecture, one agent access to leader and other agents report to it. There could be multiple levels of hierarchy as well. But the main distinction is the clear division of labor between the different sub-agents.
• In a horizontal architecture, the agents are all more or less equal and are part of a single discussion about the task. Communication between agents is shared between all of the agents. And agents can volunteer themselves to complete certain tasks or call tools.

Key Considerations

• Reasoning

  • Reasoning is basically the same thing that we humans do where we think critically about a problem, understand how it fits into the world around us, and make a decision.
  • For a model, reasoning is what allows it to go beyond its training data and learn new tasks or make decisions under new circumstances.

• Planning

  • Planning is an application of reasoning.
  • And there are five major approaches to it. Task decomposition, multi-plan selection, external modulated planning, reflection and refinement, and memory augmented planning. See understanding the planning of LLM agents.
  • Most agents have a dedicated planning step, which they run before executing any actions. There are many ways to do this. The paper particularly calls out Graph-enhanced Large Language Models in Asynchronous Plan Reasoning AKA "Plan like a graph," and tree of thought as examples which allow the agent to execute multiple steps in parallel.
    • Although my recollection of Tree of Thought was that it was more about trying different permutations of problem solving and not so much about planning.

• Tool Calling

  • Tool calling goes hand in hand with reasoning and is what really allows the model to make effective and informed decisions.
  • Many agents use some iterative process of planning, reasoning, tool calling, and then breaking up the task into further sub steps with more planning and so on.
  • But some papers point out that single agent architectures often have trouble with these long chains of subtasks.
    • https://arxiv.org/pdf/2403.03031
    • https://arxiv.org/pdf/2401.17464

Single Agent Architectures

• Proper planning and self-correction is paramount here.
• A big risk with single agent architectures is that because they don't have any external method of automatically correcting themselves, they may get stuck in an infinite loop where they reason the same step over and over again with the same result.
• ReAct
  • ReAct: Synergizing Reasoning and Acting in Language Models was one of the first single agent methods designed to improve over single-step prompting. In React, which stands for Reason Plus Act, the agent has a cycle of thinking about a task, performing an action based on that thought and observing the output.
  • Aside from improved reliability, one big advantage of this method over previous single-prompt methods was that the sequence of thoughts and actions are all there to see, so it's easier to figure out how the model arrived at its conclusion.
  • But ReAct is susceptible to the infinite loops mentioned above.
• RAISE
  • RAISE, as described in From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Models, stands for Reasoning and Acting through Scratch pad and Examples. There's no 'I' but I guess they thought RAISE sounded better than RASPE or something.
  • It's based on the ReAct method, but adds the scratchpad for short-term storage and a data set of similar previous examples for long-term storage.
    • NOTE How does this work?
  • One interesting problem that the race paper found was that agents would often exceed their defined roles such as a sales agent role which ended up writing Python code. The authors also cited problems with hallucinations and difficulty understanding complex logic.
• Reflexion
  • Reflexion: Language Agents with Verbal Reinforcement Learning
  • Reflexion is a method in which the agent is asked to reflect on its own performance with certain metrics such as success state and if the current trajectory matches the agent's desired task.
  • NOTE look at the paper to determine more about these
  • Some limitations cited by the authors
    • Reflextion is prone to falling into non-optimal local minima
    • The agent's memory is simply stored in the model's context with a sliding window and so older important items may be forgotten
• AutoGPT + P
  • AutoGPT+P: Affordance-based Task Planning with Large Language Models
  • AutoGPT+P is a technique specifically designed for use in robotics. It uses computer vision to detect the objects present in a scene. And then can use four tools to try to complete its task.
    • Plan Tool
    • Partial Plan Tool
    • Suggest Alternative Tool
    • Explore Tool
  • The model also works in concert with a traditional planning tool using PDDL or planning domain definition language. This planner helps with translating the model's instructions into things that the robot is actually able to do given its physical limitations.
  • As with many of the above approaches, it does have some problems such as sometimes choosing the wrong tools or getting stuck in loops. And at least as described in the paper, there's no opportunity for human interaction such as the agent asking for clarification or the human interrupting if the robot starts to do something wrong.
• LATS
  • Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
  • LATS is an algorithm based on Monte Carlo Tree Search. You can read the link for more details, but basically, it's inspired by Monte Carlo simulation in which you do a bunch of random runs to get a better idea of the probability space and the best action to take.
  • But as you can imagine, doing a bunch of random runs down a tree with language models can be very slow and expensive. Also, the paper doesn't tackle particularly complex scenarios.

Multi Agent Architectures

• Common themes with multi-agent architectures
  • Leadership of agent teams
  • Dynamic creation of agent teams between stages
  • Information sharing between team members
• Embodied LLM Agents Learn to Cooperate in Organized Teams
  • This method uses a hybrid approach that is mostly a horizontal team, but has a leader agent over the rest of the team.
  • They found that teams with a leader finished their tasks about 10% faster and that without a leader the agents spend about half of their time giving orders to each other. Whereas with a single designated leader, the leader spends 60% of its messages giving directions, while the other agents can focus more on actual exchange of useful information.
• DyLAN
  • Dynamic LLM-Agent Network: An LLM-agent Collaboration Framework with Agent Team Optimization
  • DyLAN is a dynamic team method which uses elimination rounds to remove the agents that have contributed the least to the task.
  • Team Optimization
    • Each agent is asked to rank the other agents’ results
    • These ratings are aggregated
    • An “Agent Importance Score” is calculated
    • Low-performing agents are removed from the system
• AgentVerse
  • AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors
  • Agentverse uses a four-stage process
    • Recruitment, which uses a “recruiter” agent to generate personas for a set of agents to work on this iteration, based on the current goal state.
    • Collaborative decision-making between the agents.
      • This can be vertical or horizontal arrangement, depending on the task.
    • Independent action execution by each agent
      • Each agent uses a ReAct loop with up to 10 iterations to get to the desired output
    • Evaluation of how close the current state is to the goal.
  • This process can be repeated until the goal is reached.
  • One important finding here is that agent feedback is not always reliable.
    • Even if an agent’s feedback is not valid, the receiving agent may incorporate it anyway.
• MetaGPT
  • MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
  • MetaGPT focuses on using structured outputs to communicate between agents instead of plain text in order to reduce unproductive chatter and inefficiencies, such as "how are you? I'm fine".
  • It also implements a message bus which allows agents to publish their information to a common place but only listen to information that is relevant to them.

Discussion

• Single agent patterns tend to work best with a narrowly defined list of tools and well defined processes. They're also easier to implement because there's only one agent and set of tools, and they don't face the limitations of multi-agent systems like poor feedback from other agents or unrelated chatter from other team members. But they are more likely to get stuck in loops and fail to make progress if they find themselves in a situation that does not match their reasoning strengths.
• Multi-agent architectures work best when feedback from different personas helps to accomplish the task, such as drafting a document and then reviewing or proofreading it. They're also useful for performing parallel execution when there are distinct independent subtasks. Multi-agent architecture is particularly advantageous when no examples of the task have been provided.
• Feedback can be very helpful, but it's not a panacea. The AgentVerse paper notes a case where an agent gave invalid feedback to another agent, but it was still incorporated. Similarly, human feedback may conflict with the desired behavior of the agent, but because the language models tend to be willing to please, they may incorporate it anyway.

• Information Sharing

  • Information sharing in a horizontal multi-agent system is very useful, but also has issues. For example, agents can too closely simulate a human when assigned a persona and start asking the other agents small-talk questions such as "how are you?" Agents may also be exposed to information that is irrelevant to their particular task, so systems that allow subscribing or filtering incoming information can be helpful for keeping an agent on task.
  • Vertical architectures tend to not have as many of these issues, but can encounter problems when the managing agent does not send enough information to its team for them to do the job. The paper recommends using prompting techniques to help with this.
• Careful design of the system prompt for the persona can help to keep an agent on task and reduce the amount of unnecessary chatter between agents.
• Dynamic team creation where agents are brought in and out of the system can be a big help because it excludes irrelevant agents from adding noise a particular stage of the problem.

Limitations

• Evaluating agents is difficult and there are not very many good standard benchmarks.
• Many papers introduce their own benchmarks alongside a new agent system, which makes it difficult to compare agent systems beyond those tested in that particular paper.
• Many agent evals are complex and require manual scoring, which can be tedious, limits the size of the evaluation set, and adds the possibility of evaluator bias. The complexity of agents also leads to a lot more variation in their outputs, so it's more difficult to properly determine if an agent's answer is correct or not.
• As with language model evaluations, data set contamination is a problem, where the tasks that the agents are trying to work on can be found in their training data.
• Many standard benchmarks designed for language model testing, such as MMLU and GSM8K, are not applicable to agents because they don't really exercise an agent's ability to reason beyond what you would find in a single call to a language model.
• Some agent eval systems use simpler answers such as yes or no, which are easier to evaluate, but this limits the real world applicability of the eval, where most tasks require more complex answers. More complex benchmarks that use logic puzzles or video games come closer, but even in those cases it's questionable how much it translates to the real world, where tasks are less well-defined and data is dirtier.
• The paper mentions WildBench and SWE-bench as a couple of benchmarks that use real-world data, though WildBench doesn't seem to be designed for agent testing.

• git checkout <BRANCH TO MOVE>
• git rebase <NEW_PARENT_BRANCH>
• av stack sync --parent <NEW_PARENT_BRANCH>

Task List

• Up Next

  • Write documentation
  • Basic UI for visualizing runs, steps, etc.

• Soon

  • Option to skip logging model messages
  • Python client
    • This should be both a normal client and have the ability to wrap other common clients such as the OpenAI SDK. Needs to be compatible with tools like instructor, DSPy, etc.
  • Add fixture tests for Fireworks
  • Add fixture tests for Together
  • Add fixture tests for Ollama
  • Add fixture tests for Anyscale
  • Add fixture tests for DeepInfra
  • When waiting to retry, detect if the request has disconnected and cancel it
  • Add test for JSON response_format

• Later/Maybe

  • Function to validate configuration and return a list of errors
  • API Management
    • Endpoints to manage aliases
    • Endpoints to manage API keys
  • Watch and reload config file
  • Request Management
    • Allow passing through user-agent header
    • Simple Request caching
      • Cache responses based on provider/model/messages
      • Support in-memory, disk, database, Redis
    • Fetch secrets from AWS secrets store
    • When looping around providers with retries, omit providers who had an unrecoverable error.
    • Option for provider-level or model-level rate limiting
      • So that new requests coming in will automatically wait or fallback
    • Monitor error rates from providers and auto-switch
      • This can be built into the alias system perhaps
  • New Providers
    • New Provider: Cohere
      • Very different API than the others, it’s different enough from a feature perspective that I’m not sure it’s worth translating between request formats.
    • New Provider: OpenRouter
    • New Provider: Google Gemini
    • OctoAI
    • Lepton
  • Specific Provider Upgrades
    • Claude: support "is_error" flag in tool results
    • Claude: support images in tool results
  • Other Modalities
    • Support binary upload APIs like Deepgram as well
  • Logging
    • Send logged data to arbitrary HTTP endpoint
      • This should be done in a way that it can sent to something like Kafka, Elasticsearch, or Clickhouse using only the configuration, no custom code
    • Send logged data to S3
      • As JSON files? As Parquet?
    • Figure out what to do about large data
      • saving input and outputs is useful but can start taking up a lot of space. Ideally we can have something that places records in cloud storage, just need to figure out the formats and so on and if/how to make things queryable before they land in storage.
  • Analysis
    • visualize by arbitrary metadata
    • Ability to create database indexes on arbitrary metadata even in JSON fields
  • Price Tracking
    • Associate each provider and its calls with a pricing plan
    • Fetch and update prices for each provider

• Done

  • Support prompt caching — Aug 25th, 2024
  • Support tools with Ollama
  • New Provider: AWS Bedrock — Jul 8th, 2024
    • https://lib.rs/crates/aws-sdk-bedrockruntime
    • https://docs.aws.amazon.com/bedrock/latest/APIReference/API_Operations_Amazon_Bedrock_Runtime.html
  • Run and Step tracing — Jun 30th, 2024
  • Add Mistral support — Jun 19th, 2024
  • Streaming with Groq and Ollama — Jun 1st, 2024
  • Enhance test suite with real-world cases — May 31st, 2024
    • This uses streaming and non-streaming responses from various provider types, for both regular text and tool calls.
  • Streaming with Claude — May 31st, 2024
  • Streaming support — May 31st, 2024
  • "Simple" API can build for Postgres
    • Dropped the "full web app" version of the API. This will come back at some later time
  • Anthropic now supports "required" tool mode
  • Recover from Groq function calling failure
  • Endpoint for generic event logging — May 8th, 2024
    • Take the same metadata that we use for LLM calls, store them in a different table with just an event type and data json blob.
  • Support tool use fields — May 1st, 2024
  • Simpler API server that supports SQLite — Apr 30th, 2024
    • This will just use the built-in proxy tables, but is better for simpler use since it writes to SQLite
    • Autoload config files from the XDG directories
  • Javascript client — Apr 29th, 2024
    • This comes with a Chronicle-specific client, and can also redirect clients such as the OpenAI SDK using a custom fetch function.
  • Submit request metadata (org/user/workflow id) via HTTP headers — Apr 29th, 2024
  • API should have default to do everything without authorization
    • Do this by not only setting up a default user, but also adding it as the anonymous fallback
  • Testing — Apr 26th, 2024
  • For API mode, add data tables as Filigree models instead of using the built-in tables
  • When multiple providers are in use, keep retrying even on normally un-retryable errors
  • Allow configuring fallback provider and model on retry. — Apr 24th, 2024
    • This is part of the model alias configuration. Basically instead of a single provider and model there's an array of provider/model/apikey tuples
  • Support model/provider aliases — Apr 23rd, 2024
  • Support api keys — Apr 23rd, 2024
    • These can only be referenced by aliases
  • Save metadata into SQLite or Postgres — Apr 22nd, 2024
  • Load model and provider definitions from a configuration file — Apr 22nd, 2024
  • Store and load model and provider definitions from the database — Apr 22nd, 2024
  • Configurable user agent for HTTP client — Apr 21st, 2024
  • Link requests to internal users/orgs/projects — Apr 21st, 2024
  • Configurable timeout — Apr 20th, 2024
  • Common format chat messages and responses — Apr 19th, 2024
  • Automatic retry with rate-limit support — Apr 19th, 2023
  • Endpoint that proxies the call — Apr 19th, 2024
  • Send all relevant metadata as Otel traces — Apr 19th, 2024

• Each price sheet entry as an active flag
• When prices are updated for a model, add a new entry and mark it active
• In the future have a scraper or other mechanism of getting latest price data for each model

• Record in a postgres table
• Output OpenTelemetry

• Org ID
• User ID
• Run ID (ID linking related prompt calls together)
• Workflow Name
• Workflow Step
• Arbitrary other metadata
• endpoint called
• provider and model used
• input text
• output text
• input token count
• output token count
• which price sheet row was used
• response time

Task List

• Up Next

  • Probably drop this and use LangGraph instead?
  • Completely serialize and restore an agent
    • This needs to account for tool context as well, shared state between nodes, nested state machines, etc.
      • Probably split up "context" and "tools" where "context" always needs to be serializable and tools can contain functions.
      • Could go more complicated with serialize/deserialize functions but this gets messy in JS.
  • Agent visualization
    • This should be both a generic application that can visualize agents in any database with a compliant schema, and also a set of components that make it easy to do so in a customized fashion.

• Soon

  • Helper function for simple chain -> DAG
  • Chatbot interface that can be adapted to web, Slack, Discord, etc.
    • Standard set of incoming and outgoing events for this case
    • Implement serialization
    • Multiple levels of verbosity
  • Serializable runner functions
    • This has the runner in the config as an object instead of a function, where the object contains a key into some runner registry that points to the function, and additional arguments to the function.
    • This will be very useful for DAGs generated by agents that may not be fixed in the code.
  • Dynamic node type that can expand its inputs into a DAG or state machine
  • Cache system
    • Mainly for tools that make network requests and for LLMs. This is mostly a debugging feature but in some cases will probably be useful for other things.
    • LLM request caching could be built into chronicle
    • Support in-memory, file-system, and network like Redis
  • Central orchestrator for multiple sub-state machines and agents.
    • This can be a convenient point for human in the loop and other types of things like that.
    • There should be some affordance here for cases where we want to allow the user to bail out of a sub-agent.

• Later/Maybe

  • Replay a DAG from any point in its execution
    • Basically just rerunning a node from its inputs and setting things up so that the descendant nodes will all run appropriately too.
  • Replay a state machine from any point in its execution
    • At each step, save the input/output and a diff of the context.
  • Update inputs and rerun only affected parts of a DAG
    • This adds a new config key to nodes, where they can indicate which keys in rootInput they care about
  • Ability to edit a node's output that wasn't what the user wanted
    • Say, if a node got the wrong result and the user wants to correct it.
    • Ability to cancel just a single node and its downstream
    • Modify output of any node and rerun downstream nodes in the DAG
    • Make "cancelled" state non-permanent. Nodes return to a ready state or something when a cancel occurs.
  • Dynamic modification of a DAG or state machine
    • Add new nodes to the existing DAG or state machine. This shouldn't be too hard to do and will help with introspection if doing dynamic planning.
    • This should also be able to take another DAG configuration and merge it into the current one, along with support for mapping or prefixing the node names so that multiple copies of a DAG can be merged.
    • Where do the outputs of the merged DAG go? Need to think more about how this makes sense and fits together.
    • A simple version of this is the node that is dynamically configured as a subgraph. This probably makes the most sense as a longer-term solutiona s well.
  • Look at integrating with https://github.com/transitive-bullshit/agentic tools
  • Short term memory system
    • Sometimes we want to do RAG or something while an agent executes but won't need most of this data afterward. This basically becomes the ability to purge and/or archive the RAG data after we don't need it anymore.
    • One-file-per-agent using LanceDB or something like that could work well here.
  • Script to parse DAG dependencies and generate a Typescript interface with proper types for the inputs and outputs of each node
  • Manage multiple related state machines
    • e.g. multiple agents running similar tasks on different input for a usre
    • or multiple parts of a single plan all executing concurrently
  • Send an event to multiple state machines based on some criteria
    • This should allow some level of arbitrary criteria, like we want to be able to select based on an organization, a user, workflow type, start time, some other tag
  • Global rate limiting of active tasks
  • Evals
    • Design conversation
    • Eval database tables
      • Eval job - name, time, source (CI, manual, etc), user, optional description
      • Eval item - a single item from an eval job that links to a run. Contains additional info around the run
      • Eval score - one score for an eval run
      • Eval data set - inputs and outputs
      • Need ability to attach extra scores by manual curation.
      • Associate metadata with a run
    • Views:
      • compare results of a particular eval across different runs
      • Results for a particular eval over time
      • Total results for a particular category of run over time (e.g. for a particular workflow, the trend of success rate)
      • All results for a run
      • Filter by whatever, including metadata values

• Done

  • Chronicle function for converting metadata into span attributes
  • Convert to multipackage monorepo
  • Allow overriding name of DAG or state machine on each run
  • Basic State machine support
    • Configuration
    • Function to execute one step of the state machine
    • Ability to call a DAG, other state machines, or an arbitrary promise-returning function in a state
    • Tracing/events on all transitions
    • Configurable transitions
    • Global error handler/state
    • Configurable override of error transition on a per-state basis
    • Send events to state machines
  • Semaphores per class of node
  • Stepwise mode for DAGs
  • Some way to send events out while still running (e.g. to send updates to the user)
  • Tracing spans for every state
  • Log state transitions, inputs, and outputs to a database
  • DAG excutor
  • Spawn state machines or DAGs from an existing task and wait for them to finish
    • This is just done via normal code, nothing special.

• "human in the loop" states
• cyclic workflow graphs
  • with some control on the number of steps taken
• Fully embrace "workflow as state machine" rather than just using them behind the scenes
• arbitrary delays
• rate limiting on how quickly an agent can make requests?

• Arbitrary delays e.g. resume after a month
• Stop to get user input, and resume
  • Sort of. This makes it somewhat easier but a simpler system can accomplish the same thing.
• Global rate limiting
• Long-running tasks so we can resume from somewhere on failure
  • This can be somewhat emulated with checkpoints or by doing a state machine inside the worker and having it save its state on every transition
• Events that can be sent to multiple dormant workflows based on some tag
  • Can be done manually though

• This simplifies the running a lot, especially for agents that don't need to sleep for a long time.
• This leaves the external component as something that can resume and run a state machine, which doesn't have to be generic at first. It can just be a normal call to the service that runs it.
• Downsides:
  • Not resilient to full process crash, although with an external sweeper this could be mollified somewhat
  • Are there any downsides around spawning sub-state machines? I think the agent software would have to be designed to allow resuming from a state like this, but otherwise it wouldn't be a problem.
• Take a look at napi-rs or neon or maybe even just in WASM, to see if we can write a lot of this in Rust. Would simplify code reuse and reliability in some ways and look forward to more full-featured plans. Might be more complex than it's worth though, need to see.
• Initial Layout
  • This is a library which can run an arbitrary state machine and persist the config and state into a database
  • Two modes
    • State machine
    • DAG
      • The difference is that because a DAG is non-looping, we can just run all the root nodes concurrently and then run other nodes as soon as their parent dependencies are done.
  • Each node in a state machine or DAG can spawn other DAGs and state machines, and wait for them to finish
  • Support running either preconfigured machine from the database or taking an ad hoc state machine
  • Some way to send events to a paused state machine to start it up again
  • State machines can run other state machines
    • Need to figure out how much explicit support this really needs
  • Ensure that we're never processing more than one state machine event at a time (except perhaps a cancel event)
  • When entering a state, supply
    • Previous state name
    • Current machine context
    • Data emitted from the previous state for this state
    • Data emitted from any submachine results that ran between the previous and current state.
  • When leaving a state, the function returns
    • Next state
    • Data for that state
    • Context updates
    • Submachines to run before transitioning, if any, and their inputs
  • Create a tracing span for every state
  • Global error state
  • States can override the global error state with some other state
  • Parallel states?
    • Need to figure out the right configuration for merging these back into a single state again though.
    • Probably easier if either
      • These are just run from within a state
        • But then it's just a normal call. Works but not great
      • These are submachines which run to completion and there's a fixed state on the other end that receives all the results.
• Library V2
  • Eventually have preconfigured blocks for certain common things, that can be called without any custom code
    • Maybe do this sooner than later depending on how agent planning goes
  • Built-in loop control with loop count limit.
    • Probably do loops as a submachine since the scoped context is useful here
  • Submachines
    • Scoped context for groups of states
    • Also useful for embedding state machines from other sources into a larger workflow.

• Probably will end up with a hybrid Moore/Mealy though

• Have some way to time out on these, like we run 10 searches with a timeout of 15 seconds and when we reach the timeout use whatever results have come back.
• Each submachine will run in parallel
• I think it's best if these child machines are actually their own separate machines which can be spawned from the state, rather than part of the parent state machine config. This is because many of these child machines will be used by multiple workflows.

• Things like "agent required correction", "this is an error", "user accepted the answer"

• Allow upserting a config where it makes a new version if the config is different from the latest version.

• This should include a list of permitted next states from any particular state

• next state. There should be some way of type checking the returned value from the worker.
• Input for the next state, specific to that state?
• Conditions to go to next state:
  • Optional delay before next transition. This can be encoded in the state machine config but also overridden
  • Some kind of polling of an endpoint?
    • Maybe the orchestrator calls an endpoint on some backoff schedule and depending on status code it either goes ot the next state or tries again. Need to think about this it might not be too useful
• Context updates

Bot Abstraction

• Keep track of conversations.
• Each "server" is a mapping of a platform (Discord, Slack, etc) and the ID of the team (Guild, Team, Org, etc.) on that platform to one of our orgs.
• Each conversation has:
  • Our own ID
  • The server it belongs to
  • Which user started it
  • The ID of the conversation in the platform. This should be JSON since it could have multiple components (e.g. a channel and thread ID in Discord) but also should be easy to look up.
    • Might be best to just use separate tables for each platform for the platform-specific part. We do still need an indication in the main conversation table of which platform a conversation belongs to so we don't have to look it up in every table.
• Sending events
  • Figure out which platform the conversation belongs to
  • Send the conversation ID and the standardized event to the platform adapter
  • Platform adapter translates it to the platform-specific format and sends it.
• Receiving an event
  • Look up the conversation, if one exists
  • Translate the event to the standard format
  • send the conversation ID and the event to the main layer

Notes from Phind

• Here are some suggestions to improve and expand on your notes for a workflow orchestrator that models workflows as Moore state machines: 1. State Machine Definition:
  • Allow defining state machines using a declarative format (e.g., YAML or JSON) for easy configuration and versioning.
  • Include additional properties for each state, such as:
    • State name and description
    • Input and output data schemas for type checking and validation
    • Error handling and retry mechanisms
    • Timeouts and maximum execution time
  • Support nested state machines or sub-workflows for modular and reusable components. 2. Step Categories and Rate Limiting:
  • Define a clear set of step categories based on the type of operation (e.g., LLM calls, database queries, external API calls).
  • Implement rate limiting at both the global and category levels to control resource utilization and prevent abuse.
  • Allow configuring rate limits per category and provide options to adjust them dynamically based on load or priority. 3. Job Priority and Scheduling:
  • Introduce a priority system for jobs to determine their execution order and resource allocation.
  • Support different priority levels (e.g., low, medium, high) and allow assigning priorities to individual jobs or workflows.
  • Implement a scheduler that takes into account job priorities and available resources to optimize execution order and minimize waiting times. 4. Callback Mechanism:
  • Provide a flexible callback mechanism for triggering state transitions and notifying external systems.
  • Allow registering multiple callback endpoints or functions for different events (e.g., state entry, state exit, job completion).
  • Support various callback protocols (e.g., HTTP, WebSocket, message queues) to integrate with different systems. 5. Context Management:
  • Enhance the context object to store and pass data between states and steps.
  • Allow defining default context values and merging them with job-specific context during execution.
  • Provide methods to update, retrieve, and delete context variables within steps. 6. Error Handling and Recovery:
  • Implement robust error handling mechanisms to handle failures gracefully and prevent workflow disruptions.
  • Allow defining error handling strategies per state or step (e.g., retry, skip, abort).
  • Support compensating actions or rollback mechanisms to revert partial changes in case of failures. 7. Monitoring and Observability:
  • Provide built-in monitoring and logging capabilities to track workflow execution, performance metrics, and errors.
  • Expose APIs or dashboards for real-time monitoring and visualization of workflow progress and status.
  • Integrate with external monitoring and alerting systems for centralized observability and incident management. 8. Scalability and Distribution:
  • Design the orchestrator to be scalable and distributed to handle a large number of concurrent workflows.
  • Implement load balancing and horizontal scaling mechanisms to distribute workload across multiple nodes or workers.
  • Consider using distributed locking and coordination mechanisms (e.g., ZooKeeper, etcd) for managing shared resources and ensuring consistency. 9. Testing and Debugging:
  • Provide testing utilities and frameworks to facilitate unit testing and integration testing of workflows and steps.
  • Allow simulating and debugging workflows in a controlled environment without affecting production systems.
  • Implement logging and tracing capabilities to aid in troubleshooting and identifying issues during development and production. 10. Documentation and Examples:
    • Create comprehensive documentation that explains the concepts, architecture, and usage of the workflow orchestrator.
    • Provide clear examples and tutorials demonstrating how to define state machines, implement steps, and integrate with external systems.
    • Maintain a collection of common workflow patterns and best practices to guide users in designing efficient and maintainable workflows. These suggestions aim to enhance the functionality, reliability, and usability of your workflow orchestrator. They cover aspects such as state machine definition, rate limiting, job scheduling, error handling, monitoring, scalability, testing, and documentation. Incorporating these features and improvements will make your orchestrator more robust, flexible, and suitable for a wide range of workflow orchestration scenarios.

• resource "aws_vpc" "my_app" {
    cidr_block = var.my_app_cidr
    tags = {
      Name = "my_app"
    }
  }
  
  resource "aws_subnet" "my_app" {
    vpc_id = aws_vpc.my_app.id
    cidr_block = var.my_app_cidr
    # Set appropriately for your needs
    map_public_ip_on_launch = true
    availability_zone = var.az
    tags = {
      Name = "my_app"
    }
  }
  
  

• You can also use a NAT gateway or something, but this is the simplest and cheapest way to go.
• Here we'll also set up a VPC endpoint to link directly into S3, which saves egress charges for going through public routes.

• resource "aws_internet_gateway" "my_app" {
    vpc_id = aws_vpc.my_app.id
  
    tags = {
      Name = "my_app"
    }
  }
  
  resource "aws_route_table" "my_app" {
    vpc_id = aws_vpc.my_app.id
  
    route {
      cidr_block = "0.0.0.0/0"
      gateway_id = aws_internet_gateway.my_app.id
    }
  
    tags = {
      Name = "my_app"
    }
  }
  

• resource "aws_vpc_endpoint" "my_app_s3" {
    vpc_id       = aws_vpc.my_app.id
    service_name = "com.amazonaws.us-west-2.s3"
  
    tags = {
      Name = "my_app_s3"
    }
  }
  
  resource "aws_vpc_endpoint_route_table_association" "my_app_s3" {
    vpc_endpoint_id = aws_vpc_endpoint.my_app_s3.id
    route_table_id = aws_route_table.my_app.id
  }
  
  

• **/node_modules
  **/*.js
  **/*.ts
  **/apps
  **/target
  !apps/my-app
  apps/my-app/target
  

• #!/bin/bash
  docker build \
    -t $NAME \
    -f Dockerfile \
    --ignorefile .dockerignore \
    ../../ # The monorepo root
  

Building for Rust

# This Dockerfile works with cargo chef, which prebuilds dependencies in a
# separate Docker image to speed up builds.

FROM lukemathwalker/cargo-chef:latest-rust-1 AS chef
RUN apt-get update && apt-get install -y pkg-config libssl-dev
WORKDIR /app/apps/my-app

FROM chef as planner
COPY ./libs/some-lib /app/libs/some-lib
COPY ./apps/my-app /app/apps/my-app
RUN cargo chef prepare --recipe-path recipe.json

FROM chef as builder
COPY --from=planner /app/apps/my-app/recipe.json recipe.json
# Build dependencies - this is the caching Docker layer!
COPY ./libs/some-lib /app/libs/some-lib
RUN cargo chef cook --release --recipe-path recipe.json
# Build application
COPY ./apps/my-app /app/apps/my-app
RUN cargo build --release --bin my-app

FROM debian:bookworm-slim as runtime
RUN apt-get update && apt-get install -y pkg-config libssl-dev ca-certificates
RUN update-ca-certificates
COPY --from=builder /app/apps/my-app/target/release/my-app /usr/local/bin
ENTRYPOINT ["/usr/local/bin/my-app"]

• You need to create an ECS cluster but it doesn't need any configuration beyond being created.

• resource "aws_ecs_cluster" "pipeline" {
    name = "pipeline"
  }
  

• Add "runtimePlatform": { "cpuArchitecture": "ARM64" } to your task definition.

• A task can have a task role and an execution role.
• The task role is given to your containers.
• The execution role is given to the instance that runs your containers.
• Both of these need an "assume role policy" that allows ECS to assume those roles.
• Terraform for the roles. This also sets up extra S3 permissions on the task execution role in case you are sending your logs to S3 (see below).

  • data "aws_iam_policy_document" "ecs_assume_role" {
      statement {
        actions = ["sts:AssumeRole"]
        effect = "Allow"
        principals {
          type = "Service"
          identifiers = ["ecs-tasks.amazonaws.com"]
        }
    
        condition {
          test = "ArnLike"
          variable = "aws:SourceArn"
          values = [
            format("arn:aws:ecs:%s:%s:*", var.aws_region, var.aws_account_id)
          ]
        }
    
        condition {
          test = "StringEquals"
          variable = "aws:SourceAccount"
          values = [
            var.aws_account_id
          ]
        }
    
      }
    }
    
    resource "aws_iam_role" "my_task_execution_role" {
      name = "fargate_my_task_execution"
    
      assume_role_policy = data.aws_iam_policy_document.ecs_assume_role.json
    
      inline_policy {
        name = "s3_put"
        policy = jsonencode({
          "Version": "2012-10-17",
          "Statement": [        
            {
              "Effect": "Allow",
              "Action": [
                 "s3:*"
              ],
              "Resource": [
                "arn:aws:s3:::my-app-logs",
                "arn:aws:s3:::my-app-logs/*"
               ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "ecr:GetAuthorizationToken",
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
              ],
              "Resource": "*"
            },
            {
              "Effect": "Allow",
              "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:DescribeImages",
                "ecr:DescribeRepositories",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:Images"
              ],
              "Resource": [
                format("arn:aws:ecr:%s:%s:repository/*", var.aws_region, var.aws_account_id)
              ]
            }
          ]
        })
      }
    }
    
    resource "aws_iam_role" "my_task_role" {
      name = "fargate_my_task"
    
      assume_role_policy = data.aws_iam_policy_document.ecs_assume_role.json
    
      inline_policy {
        name = "s3_put"
        policy = jsonencode({
          "Version": "2012-10-17",
          "Statement": [{
            "Effect": "Allow",
            "Action": [
              "s3:*"
            ],
            "Resource": "*"
          }]
        })
      }
    }
    
    output "my_task_role_arn" {
      value = aws_iam_role.my_task_role.arn
    }
    
    output "my_task_execution_role_arn" {
      value = aws_iam_role.my_task_execution_role.arn
    }
    
    

• Cloudwatch is the AWS-recommended way to ship logs out but you can also send them to S3.
• First you'll need roles like the ones in the previous section to give S3 permissions.
• Then create your S3 bucket.
  • This configuration also autodeletes the files after 90 days.

  • resource "aws_s3_bucket" "my_app_logs" {
      bucket = "my-app-logs"
    }
    
    resource "aws_s3_bucket_lifecycle_configuration" "my_app_logs" {
      bucket = aws_s3_bucket.my_app_logs.id
      rule {
        id = "delete-old-logs"
        expiration {
          days = 90
        }
        status = "Enabled"
      }
    }
    
    

• Then set up your task definition like so.

• {
    "taskRoleArn": "ARN of the task role above",
    "executionRoleArn": "ARN of execution role above",
    "requiresCompatibilities": ["FARGATE"],
    "containerDefinitions": [
      // Only needed for custom log routing.
      {
        "name": "log_router",
        "essential": true,
        "image": "amazon/aws-for-fluent-bit:stable",
        "firelensConfiguration": {
          "type": "fluentbit"
        },
        "logConfiguration": {
          "logDriver": "awslogs",
          "options": {
            "awslogs-group": "firelens-container",
            "awslogs-region": "us-west-2",
            "awslogs-create-group": "true",
            "awslogs-stream-prefix": "firelens"
          }
        },
        "memoryReservation": 50
      },
      {
        "name": "my_app",
        "image": "acctnum.dkr.ecr.region.amazonaws.com/image:tag",
        "portMappings": [],
        "essential": true,
        "logConfiguration": {
          "logDriver": "awsfirelens",
          "options": {
            "Name": "s3",
            // Optional, to customize key format which can be useful for batch jobs
            "s3_key_format":
              "/%Y/%m/%Y-%m-%d-my-app-$TAG-%S",
            "region": "your-aws-region",
            "bucket": "my-app-logs",
            // Rotate to a new log file when this size is reached.
            // For persistent services this should be much smaller.
            // For batch jobs I set it large since it's nice to have
            // all the logs for a run in one file.
            "total_file_size": "10G",
            "upload_timeout": "1m",
            "retry_limit": "2"
          },
        }
      }
    ]
  }
  

• The logConfiguration options are documented at https://docs.fluentbit.io/manual/pipeline/outputs/s3

Task List

• Up Next

  • Figure out what to do with this project
    • I'm still not getting something I really like with SQL generation, but ORMs are still too inflexible. What's the proper solution here?
    • Maybe
      • Rework auth to use JWT method
      • Autogenerate Drizzle table definitions for database so that the frontend code can just use Drizzle for database access.
      • Maybe use GraphQL for complex object fetches?
  • Make test objects easier to manage
    • Instead of a one-size-fits-all bootstrapping we should just do it with a custom function per file, which will speed up the tests overall a bit, and also greatly simplify testing of through models where we need to make sure that the IDs all line up between the different objects.

• Soon

  • Investigate making it easier to auto-forward page actions through SvelteKit to the API
  • Finish tests for through relationships
    • Do this after Make test objects easier to manage
  • Evaluate faster API authentication methods between SvelteKit Backend and Rust Backend
    • Basically to make it work better to make multiple API calls from a single request without hitting the database every time to look up the user/roles/etc.
    • Instead of just a session ID, have an option to generate a short-lived JWT or similar token that contains all the auth info. Store the JWT in the cookie, on each request check the expiration time and get a new one if getting close to expiring.
    • Might also need some hook for client-side requests that can do the renewal if they talk directly to the API.
  • Use docker compose with separate containers for API and SvelteKit parts
    • More trouble than it's worth to force the two parts into a single container
  • More flexible configuration
    • Switch from clap to config or something for the serve command's config
    • Main impetus here is to make it possible to read configuration from ConfigMap-sourced volumes in K8S and similar systems.

• Later/Maybe

  • Simplify permissions system when using custom auth
    • Since we aren't creating the roles and users, we need a way to make sure that people have reasonable permissions when they start.
    • Perhaps just a function that adds default permissions or something like that when we first see a user.
  • organization_members needs to be extendable like users is
    • This is so that there's a place to attach per-org metadata and fields
    • I think the proper thing to do here is that once we fully support Joining tables — Jul 22nd, 2024 we can just put this into the system as a joining table.
  • Improve API error printing
    • Some errors use source and error_stack's default printer ignores it. Should walk the source chain as well. This seems difficult though without using nightly features
  • Metrics tracking and alerts
    • Consistently high CPU, RAM usage, etc.
    • Might just use a service for this, need to see
  • Option to only allow connections from certain IP ranges
    • eg https://www.cloudflare.com/ips/ when using Cloudflare
  • Helpers for retrying requests and other operations
    • rate limit retry-after
    • exponential retry with jitter, etc. for regular retryable failures
    • ability to not retry for certain errors
    • Maybe just fork backon for this?
  • QoL Issues Encountered
    • Conflict in src/main.rs on first run
      • Maybe detect if this is the first run by looking at absence of .state directory and ask if it's ok to just replace it
  • Add auto-bootstrap option
    • When appropriate environment variables are set, allows the API to do the bootstrapping itself when it first starts up
  • Ops helpers
    • This isn't part of the core framework, but a set of tools for automatically doing ops-related tasks.
    • Back up database to cloud storage on a regular basis
    • Deploy on Git push
    • Preview environments?
    • Set up Postgres slow statement tracking and create a report on what was slow
    • Drop partitions of old data
  • Frontend
    • SvelteKit: Error page
    • Add dependencies to package.json like we do for the Rust side
    • App shell with nav bar, etc.
    • Password reset page
    • Profile management page
    • Manage your organization
    • Default Model editor page
    • Default Model list page
    • Admin: invite users to app
    • Admin: administer organizations
    • 404 page
    • TS Type Enhancements
      • Permission names per model
  • HTMX support improvements
    • Debugging option to show a toast whenever a livereload finishes or an htmx error occurs
    • move client-side code from SBBP into Filigree for reuse by other projects
    • theming, dark/light mode with persistence via localStorage and header script to inject dark class
      • DaisyUI provides a lot of this already
    • Island architecture support for Svelte components
      • Each of these should be its own entry point in the Vite config, and from there we can use the manifest support to get the necessary tags
    • Flash/toast support
      • Can probably be done with HX-Trigger header, or maybe an extension
      • Adapter function for actions to have errors return a toast
    • Implement fancy dropdown menu in Alpine with floating-ui
      • https://github.com/awcodes/alpine-floating-ui may work well
      • https://alpinejs.dev/plugins/anchor is the official one, seems to support fewer options
    • Improve livereload performance
      • Not sure how much opportunity there is here but it feels like it can be made 20% faster
      • Dioxus is working on a solution that I should be able to adapt once it's public.
  • Models Backend
    • Move a lot of the auth queries like add_permissions_to_role into the template system, so they can be placed on the model and/or customized
      • Everything in fiiligree/src/users except add_user_email_login which is used by the OAuth system and so is harder to move.
    • Generate endpoints for grandchild objects
      • Probably need to make this generally recursive which may difficult to do via the current template system, might need a macro that can call itself or some fancier fragment rendering.
      • I think what we want to do here is for each item in children to potentially have its own children, instead of recalculating each one independently. Also need some way to walk up the stack of children here for URL generation (or do we? Probably it doesn't matter to have the grandparent in the URL itself).
    • Make it easier for differing get/list model types to be used interchangeably
      • This matters more for HTMX mode
      • Basically we might have a function that renders something for a particular object, but it gets difficult when the input can be either the Get or List result types when they are different.
      • One option may be to generate a type of view struct which can take a reference to either type and then generate a bunch of methods for the shared fields which can return either value. This is a lot of boilerplate, but its all autogenerated, but these aren't used too often
    • Allow joining models to have their own extra data fields
    • For runtime queries, create query builders that use SeaQuery to add the auth clause automatically
      • Add an enum of columns in the table
      • Allow adding any population of data or id at runtime
      • Each child population can be seen as a column as well (but probably don't do this, it should )
      • Add functions for "all base columns" to drop into the query
    • tests for list endpoint filtering/pagination/sorting
    • Models with non-writable, but required, fields don't generate valid tests
      • This can happen when the normal "create" workflow happens in some other way than calling a REST create endpoint.
      • Also consider detecting this and just not generating an "insert" script that can't work.
        • The criteria is a model that has at least one field which
          • is not fixed
          • is not nullable
          • has no default (SQL or Rust)
          • is not writable by owner
    • Support file uploads in multipart form submissions
      • A lot of the underlying framework for this is ready but the code itself needs to be written. Main issue is that the current FormOrJson can't be used in that case. Instead we probably need something that even on a JSON file looks like a multipart submission, but just doesn't return any files.
    • Test populated get and list when updatewithparent is false
    • Models can define extra permissions
  • CLI / Template System
    • Bug: CLI doesn't throw an error if a field is defined twice
    • Use toml_edit to allow adding features to Cargo.toml
    • When generating migrations, read in the actual app migrations and ignore tables that we don't care about.
      • This lets us see things like model tables having been removed if, say, a migration file was deleted since it did the wrong thing.
      • Do need to figure out how to handle extra columns added in the model via manual SQL migrations here.
    • Command to print config and diff files against last gen and current config gen
    • Consider using something like sea_query to generate queries instead of a bunch of fixed queries
      • Will add flexibility, and could make a lot of things simpler. Need to see if there's still a good way to pre-generate queries when running the CLI
    • Data-driven default org contents, even if just a JSON blob to start
      • This can piggyback off of the fixture system
      • Considering not doing this since it may be simpler to just do it in a function.
    • Some way to easily add hooks to CRUD operations
      • Developer should probably just edit the generated code here.
      • Maybe add stub functions to each model to allow adding pre/post save hooks
    • Make it easy for additional queries to use the permissions macros
      • Probably moving away from SQL and toward Rust code for permissions checks so this won't matter as much
      • Can suggest using sqlweld here
      • See if there's a path to using a query builder like sea_query instead, maybe as an addition to the existing thing.
    • Autogenerate belongs_to field if necessary when the inverse has is detected without a through table.
      • Currently this returns an error instead
  • Authentication and Registration
    • Make organization_members a real model instead of just a table
      • Once joining tables work properly
    • Make SessionBackend work with custom auth
      • It mostly works but when the tables are in another schema then it doesn't. Need a less-fixed query method here.
    • Rate limiting for login-related endpoints
      • https://docs.rs/governor/latest/governor/
      • https://lib.rs/crates/tower_governor
      • https://github.com/brandur/redis-cell if i need this to be distributed
        • DragonflyDB is a redis-compatible that has this built-in
    • Passwordless login should include a code that can be pasted into the UI to log in instead.
    • Password requirements
      • length
      • mix of character types
    • Prevent reuse of recent passwords
    • Expire passwords that need a rehash due to upgraded standards
    • Option to expire passwords after a certain amount of time
      • This isn't recommended practice but some enterprise clients require this in order to sign with you.
    • Passkeys
      • https://passkeys.dev/
      • https://github.com/1Password/passkey-rs
    • RSA timecode Authenticator app
    • Authenticator Account recovery codes
    • Account self-deletion
    • Optional TOS acceptance at registration
    • Service accounts
    • Record info about sessions (geoip, user-agent, etc.)
    • Allow viewing other sessions for your user, and potentially logging them out
    • Invite users to your team
    • Invite users to the app, but not in your team
    • Link a new OAuth login that may not have a matching email to an existing account
    • When logging in via OAuth, fetch all known emails from the OAuth provider and see if any of them match an existing account if we haven't seen the account before.
      • Need to handle the case where there are multiple emails and they match different accounts. This probably requires an interstitial after the login page to select the matching account.
    • Waitlist functionality
    • Put auth code behind an adapter so it can be swapped out for a 3rd-party service
  • Permissions and Authorization
    • Project-based Access Control
    • Consider separating permissions checks from the SQL queries themselves
      • This has the upside that it would simplify a lot of checking for permissions and make more complex permissions easier
      • Upsides
        • SQL templates become simpler
        • Much easier to implement ABAC
        • Easier to differentiate authz errors from missing objects
      • Downsides
        • Requires remembering to actually check for permissions, though that's alleviated some by the template system since the checks can be placed in the access functions.
        • We also end up reading objects multiple times in update/delete situations, though this isn't that big of a deal. For project-based permissions we could potentially load all the projects though would have to see about that.
      • Should probably do this as permissions is becoming more and more complex as more options are added.
      • Maybe consider something like casbin or biscuits too. Feels like overkill at this point for most things though, probably more appropriate for an API-first product.
      • list query would still need permissions checks in the SQL
    • API Key lookup query needs to change to make permissions a subset of the user's permissions when inherituserpermissions is false
      • This way if a user has some permissions removed, they can't bypass those limits using an old API key which still had those permissions
      • This is actually a good use case for casbin or biscuit, though could still be implemented without it.
    • Feature flags
    • Add "all" model permissions so that some roles have permissions on everything even when new models are added.
      • This needs some more thought though. Some models like user, org, etc don't want to have this blanket rule applied. Might be better to just do this via migrations.
      • Better: Have some models use global org-wide permissions of just "read," "write," etc. and only specific models have their own set of permissions.
    • Ability to apply limits on how many objects are created
      • There's a lot of flexibility here, since the limits could be something simple like number of objects, or some derived metric such as "number of minutes" for an audio-based model.
      • The source for the limits could also come from a number of different places, most commonly pricing plan but could be other stuff such as number of community contributions, upvotes, etc.
      • So the question is what's the proper framework to make it easy to write code that accomplishes all this.
        • Make it easy to fetch the limits data.
        • Make it easy to enforce rules around these limits.
    • API call quotas
      • e.g. 3000 calls per month, 20 calls per hour
      • This should be able to group endpoints by categories where they all share a quota
      • Might need to do this with a combination of event tracking and a rate limiter
    • Usage-based pricing
    • Ability to add rate limiting on any endpoint
      • keyed by org
      • keyed by user
      • endpoints can share a bucket
      • Local tracking is fine at first, should eventually support doing it in Redis or similar to support multiple API servers.
    • Ability for admins to assume permissions of other users
      • This needs to be audited as well even if nothing else is audited
  • Workers
    • Ability to run worker tasks in some other process/VM/FaaS
      • Need to figure out configuration for this... this might be better implemented in the task queue itself
    • Outbox pattern support for transactional starting of background jobs
      • Not important for now since Effectum only runs in-process but at some point it will have a server mode and then it will make more sense.
  • Block Storage and File Uploads
    • Option for Uploads via signed URL
    • Generate tests for file upload endpoints
    • When retainfileon_delete is set, optionally add an entry to a table that lists orphaned files.
      • The idea being that we might want to hold on to orphaned files for some period of time, and then delete them later.
    • Image statistics on upload
      • format, width, height
        • Can detect on upload using code from Pic Store
    • Generate image thumbnail and blurhash
      • Best done in background worker
    • Potentially need a way to upload a file before the parent object exists. Will probably wait for a real use case on this one.
      • Could make the parent ID nullable here and then add an endpoint to "claim" a file for a particular parent object
      • Also could just do the Github style thing and upload them to a publicly available bucket. We don't maintain a file database entry in this case, and just return the public URL for whatever use it is.
    • Allow specifying storage provider presets from env vars, not just in config
  • Notifications
    • Notifications: Other types of notifications?
    • Notifications Internal notifications
      • e.g. Discord webhook notifying myself about events
      • This should have notification templates and also destinations for the notifications
  • Event logging
    • Should support arbitrary types of events and also allow setting up queries that filter on various event sequences for a user/org
  • Workflow System
    • Build the workflow system
      • A lot from Ergo can be reused
    • State chart implementation
    • Workflows can "sleep" by enqueueing a scheduled task that will continue running when it wakes up
    • Endpoint to send an event to a workflow
    • Define workflow actions
    • Ability to define workflows in the app config
    • Scripting for Workflows
      • Piggyback on Ergo but maybe switch to QuickJS or saghul/txiki.js: A tiny JavaScript runtime
  • Hosting
    • Not sure if I'll do any of this, it's really better handled by a reverse proxy
    • LetsEncrypt support
    • TLS/HTTPS support
    • HTTP -> HTTPS redirect
  • User model list/get functions need to return users which are in multiple orgs, and whose current org is a different one
  • Easy setup for commercialization
  • Collect and export metrics to some tool
    • Looks like Honeycomb supports Prometheus format
  • Basic Analytics
  • More complex workflow/funnel analytics
  • Data Hooks and Similar
    • Delete log table for possibility of undelete later
    • Option for auditing on all operations
  • Password confirmation flow for "dangerous" actions
  • User profile photos
  • Copy all tests in test-app into filigree crate where it makes sense
    • Some tests are more convenient to test in the test app, but rely on minimal scaffolded functionality. Should add a minimal server/migration setup in filigree crate tests so that they can run there as well.
  • Support SQLite
    • Not sure how much work it will be to support both Postgres and SQLite, could turn out to be a big hassle due to differing feature support and syntax, but I'll see at some point
    • Things I use that are different in SQLite
      • AL JOIN (it supports them, just not with this syntax)
      • No native uuid and timestamptz types
      • No array_agg - this is probably the biggest one
      • jsonb functions are a bit different
      • parameter binding syntax is different

• Done

  • Use a build script to inject the current filigree version into the value used by add_deps — Aug 6th, 2024
    • Minor thing but makes the release process a bit smoother. Right now this has to be updated manually.
  • When using "string auth IDs" or custom auth in general, make it a different field that references the user/organization/role ID in the external system, and continue using a normal ObjectId type internally — Jul 22nd, 2024
    • This simplifies a whole bunch of code that otherwise has to go back and forth depending on the actual underlying type of the auth object.
  • Joining tables — Jul 22nd, 2024
  • Model should have functions for adding/updating/removing child objects — Jul 17th, 2024
  • Rework query template system — Jul 16th, 2024
    • Create more of the query template stuff in the Rust code instead of in Tera
      • This will make certain things a lot simpler especially when dealing with binding parameters
      • When generating a static query, keep a mapping of binding parameter to position, and have some method for the template system to generate a bunch of bindings from an object.
        • This allows the query system to call a single function with data generated along with the bindings, which adds the relevant parameter bindings and doesn't require us to keep the two places in sync.
    • Remove _permission from all the structs
      • This removes the ability to have the custom Serialize impl that only serializes fields visible to the Owner if the permission is write.
      • How useful is it anyway to have fields which are only viewable by the owner? Maybe it doesn't matter.
    • Remove permissions checks from queries for global auth scope cases where we already know the permissions from the roles
      • project-level and object-level still need it
    • Change all the query functions into objects on a struct. Consider making the struct into a thing that takes a reference to the AuthInfo.
      • This mostly just makes the typing experience nicer since instead of post::queries::create you call Post::create.
  • Make user/org/role/etc. models optional — May 26th, 2024
  • Allow user/org/role IDs to be strings when using custom auth — May 26th, 2024
  • Add more test apps — May 24th, 2024
    • Add more as big new features are added
    • These should mostly be .gitignored except for the files that filigree doesn't generate
  • Bug: Unique fields need to be unique per org, not globally — May 23rd, 2024
  • Pluggable auth system — May 22nd, 2024
    • Support talking to some other auth microservice, external service, using JWTs or Biscuits, etc.
    • This requires
      • A new function on the AuthQueries trait that can take a request Parts instead of a pre-parsed API key or session ID
        • Or maybe just change the api_key to pass the raw Bearer token, and make it easy to unpack that into an API key for standard uses
      • trait functions should return some error type other than an sqlx::Error. An AuthError is probably the way to go or maybe even just start using Report there so we can save the internal error
  • Interactive bootstrap command — Apr 26th, 2024
  • Command to list all the environment variables that will be read along with their defaults if not provided
    • Might be a bit complex but worth it since there can be a lot
  • Bug: when deleting a model the files under src/model/NAME/ are not removed.
    • Probably need to read the .state directory to see what files we generated before but aren't being generated this time
    • Also don't delete files which have been edited
  • Anonymous user mode
    • Anonymous users get a default Authed with a specific org and user ID created just for the anonymous user.
  • Allow specifying a user ID for anonymous users to impersonate. This lets them have specific permissions, see certain objects, etc. — Apr 16th, 2024
  • Add init command that runs Cargo init, create-svelte, and creates basic config.toml? — Apr 16th, 2024
  • Add a command that creates a new Postgres user, password, and database for your project — Apr 16th, 2024
  • Support maud/htmx as a SvelteKit alternative for less complicated apps — Apr 16th, 2024
  • Switch out deprecated Jaeger tracing crate for opentelemetry-otlp — Apr 15th, 2024
  • When a type and its populated type are the same, don't generate e.g "ListAndPopulatedListResult", just "ListResult" — Apr 15th, 2024
  • htmx: manifest alterations need to be a vite plugin so they work in dev mode — Apr 12th, 2024
  • htmx: live reload — Apr 12th, 2024
  • htmx: asset pipeline config to bundle CSS and scripts, compress assets, etc. — Apr 11th, 2024
  • htmx: build JS with vite or esbuild, read from file manifest to get list of static files with hash — Apr 11th, 2024
  • htmx: Simple bundling with vite — Apr 11th, 2024
  • htmx: Read vite manifest to support hashed values — Apr 11th, 2024
  • htmx: Instantiate manifest object in app, set up watcher in dev mode — Apr 11th, 2024
  • htmx: obfuscate error middleware should only run on JSON responses (although maybe do HTML obfuscation too) — Apr 10th, 2024
  • htmx: Potentially a new extractor that lives alongside FormOrJson that always returns the form — Apr 10th, 2024
  • htmx: Customize / route in pages — Apr 10th, 2024
  • htmx: Tailwind CSS support — Apr 10th, 2024
  • Auth failures need to redirect to login page instead of just returning a 404 error - — Apr 9th, 2024
    • Errors in general need to do that
  • Skip rendering sveltekit files when it is not enabled — Apr 8th, 2024
  • Remove "sync types" and TS generation code when not using sveltekit — Apr 8th, 2024
  • htmx: Page wrapper that gets standard <head> tag contents for all — Apr 8th, 2024
  • htmx: Allow defining pages and actions — Apr 8th, 2024
    • This works similar to the existing endpoints stuff but made for HTML app
    • Autogenerate a rust module for each path and its actions
    • Should this be in a nested directory structure or just by filenames? Maybe nested directories
  • Backend Error reporting — Apr 7th, 2024
    • Almost done, just need to hook up the error reporting interface into FiligreeServerState so that I can report errors outside of the automatic system.
  • Frontend error reporting — Apr 7th, 2024
  • Generate JS code to talk to the API
    • This is done for custom endpoint functions, not yet for CRUD
  • Add configuration to enable trace export — Apr 6th, 2024
  • Add SQL functions to go between an object ID text and UUID forms. — Apr 5th, 2024
    • Should have these left over from Ergo
  • Frontend: Generate client-side validation — Apr 5th, 2024
    • This is mostly done with the generated zod schema, but some things are needed such as checking against create or update schema as appropriate
    • Right now the UI doesn't update right on a client-side validation failure
  • Set it up to work in production — Apr 5th, 2024
    • Dockerfile
    • handleFetch hook
    • Rust side has a fallback route which directs data to the SvelteKit server
      • Most uses will also have a reverse proxy in front but this will let it work without that
  • Command to print all the environment variables that the generated application will check — Apr 2nd, 2024
  • Specify custom endpoints per model — Mar 29th, 2024
    • This generates an endpoint function, a route, path/query/body parameters, input/output/query structures, and corresponding Typescript functions and interfaces to use it.
  • Allow omitting certain normal fields on list — Mar 27th, 2024
    • e.g. if there's a large JSON field not needed for the list page we should omit it.
  • Easy generation of TS types from Rust types — Mar 25th, 2024
  • Background jobs v1 — Mar 23rd, 2024
    • Define background jobs in config
      • Option to set recurrence in config
    • Global and per-job concurrency settings
      • Actually do this via multiple worker configurations. Can configure workers equal to your concurrency desires.
    • Generate a payload structure for the background job
      • No need to define the contents in the config, the user can just fill in what they need
    • Generate a helper to enqueue a background job
    • Generate a stub payload and function that runs the job
    • Hook it all up
      • Add queue to ServerState
      • Make environment variables
  • Option to allow setting an object ID when creating an object — Mar 22nd, 2024
    • This can help with scenarios where we want to create an object and potentially write it to the database later, but have the ID to use beforehand.
  • Generate Typescript interfaces for models — Mar 19th, 2024
  • Generate tests for child object url endpoints — Mar 17th, 2024
  • File model type — Mar 15th, 2024
    • Files are a separate model, and when they need to be referenced by another model they can just form a parent-child relationship.
    • Add a function for upload
      • where should this function go? probably a new file which handles upload, delete, and whatever else might be necessary
    • Add an endpoint to process an upload directly
    • Delete file when model is deleted
    • Delete files when parent model is deleted
    • Special file configuration
      • upload path template
      • a bucket from storage.bucket to store an upload
      • permissions for download or not
    • Extra optional fields
      • hash
      • file size
      • original filename
  • Initial Block Storage Support — Mar 10th, 2024
    • Block storage abstraction
      • use object_store crate for the underlying interface
    • Storage configuration
      • The idea here is that each storage location can be configured but the details should be completely configurable at runtime.
        • Or should it always be environment based? Probably yes actually except maybe some basics like bucket name, endpoint, etc.
      • something like STORAGE_${location_name}_BUCKET and so on for each provider
    • Use storage configuration in template
    • Stream request body into object storage
    • Stream multipart files into object storage
    • Allow examining data as it streams through
    • Stream file from object storage as Response body
  • Create endpoints should return children when doing updatewithparent — Feb 29th, 2024
  • Store .gen files in a tar file instead - Cancelled Feb 28th, 2024
    • This reduces the huge number of files in the state directory, but makes merge conflicts more difficult to deal with.
  • Parent/child model relationships — Feb 27th, 2024
    • Configuration
    • Add fields for belongs_to relationships to model structures
    • Generate composite primary key for joining tables and update queries appropriately
    • handle has_one relationships in fetch queries with ID fetch
    • handle has_one relationships in fetch queries with data fetch
    • handle has_many relationships in fetch queries with ID fetch
    • handle has_many relationships in fetch queries with data fetch
    • Allow populating references fields on fetch when configured
    • Updates for has_one relationships with data payload
    • Updates for has_many relationships with data payload
    • Generate CRUD endpoints for children of a particular object
      • This is basically the same as the normal endpoint, but with the parent ID specified in the URL instead of in the payload.
      • When it's not a many relationship, omit the child ID parameter
    • Populated query functions
      • Call populated functions from endpoints when appropriate
    • SQL Queries for child objects
      • List by parent ID
        • This could be done as just another filter on the list query instead of a new query template
      • insert/update multiple objects
      • Delete by parent ID where ID is not in the set of IDs just inserted/deleted
    • Ensure that each model is not referenced my more than one non-through has relationship
    • Sort CREATE TABLE statements so that parents come first
    • First round of Tests
      • Create a test model with children
        • Post
        • has one Poll
        • has many Comments
        • has many Reactions
      • populated queries with has_one
      • populated queries with has_many
      • insert with child object
      • update with child object
    • create and update functions should return the resulting objects
  • Test passwordless login in Glance — Feb 14th, 2024
  • Test login with password in Glance — Feb 13th, 2024
  • Frontend: Handle validation errors from server — Feb 11th, 2024
  • On validation failure, return the sent data
    • This helps when SvelteKit has forwarded the request straight to the server, since it allows Kit to return the form data back again to repopulate the form
  • Command to bootstrap initial org and user — Feb 11th, 2024
    • This should have the ability to take admin user's email from either the command line or the environment
  • Initial OAuth login support — Feb 10th, 2024
    • Update createnewuser to implement UserCreator trait
    • Allow creating a user without an email address
    • Look for client id/secret environment variables for each provider and create the ones that are found — Feb 3rd, 2024
    • Hook up oauth code to endpoints — Feb 3rd, 2024
    • Link to existing account when it's a new OAuth login but a known email — Feb 2nd, 2024
    • Client-side code for OAuth login — Feb 9th, 2024
  • Login page — Feb 10th, 2024
  • Easier way to add new fields to base models than redefining the whole model — Jan 29th, 2024
  • Integrate into Glance
  • User Creation updates — Jan 27th, 2024
    • Add default role field to org, for users who are added without any set of roles
    • Add roles field to email_invites
  • Validation for form submissions — Jan 22nd, 2024
  • Use hosts list in post-login redirect processing
  • JSON validation that checks multiple fields
    • do this using axum-jsonschema
    • Derive schemars::JsonSchema on every structure that might need to be validated
    • Might want to add additional validation in the future with an extension to this
  • CORS configuration — Jan 17th, 2024
    • Host allow list for apps that want to lock to the app's host
    • Use CorsLayer::permissive() on /api for apps that want to expose the API to browsers
  • Initial SQL migration should only be written once — Jan 17th, 2024
  • New models should be added in new SQL migrations — Jan 17th, 2024
  • Column changes to models should generate new SQL migrations — Jan 17th, 2024
    • sql-migration-sim crate is written, just need diff functionality
    • Add index tracking to sql-migration-sim
  • New runs diff against previous run and only apply diffed changes — Jan 16th, 2024
    • When running the CLI, save the generated templates (post-formatter) to a state directory
    • On each run, diff the generated templates against the previous state in that directory
    • Apply that diff to the actual output file
      • Need a patch style that matches against nearby lines instead of just using line numbers.
    • Point out conflicts as they occur, probably use git style merge conflict markers
    • diffy supports three-way merge so that's also worth looking at
    • This system allows the user to make changes to the generated files but still update the config.
    • Need a command to rewrite a particular file
    • Need a command to just regenerate the state cache, for when changing formatter settings
  • Ability to make signup open to the public or admin-only — Jan 12th, 2024
  • signup endpoint to create a new org that also creates default roles, adds a user, etc. — Jan 12th, 2024
  • ability to sign up via passwordless email auth if enabled — Jan 12th, 2024
  • Password forgot/reset endpoints — Jan 12th, 2024
  • Passwordless email auth — Jan 11th, 2024
  • Email template system — Jan 11th, 2024
  • List all permissions in a file with descriptions — Jan 9th, 2024
  • For models with global permissions, add permissions check in route layer before hitting database — Jan 9th, 2024
  • MVP Basic email sending, enough to do password management and such — Jan 9th, 2024
  • Integrate with email sending services — Jan 9th, 2024
  • Endpoints for retrieving and updating your own user — Jan 7th, 2024
  • Auto-add appropriate dependencies to Cargo.toml — Jan 7th, 2024
  • Tests for users without proper permissions for each action — Jan 6th, 2024
  • Generate tests for models — Jan 6th, 2024
  • Tests for login/logout — Jan 5th, 2024
  • Generate fixtures for tests — Jan 5th, 2024
  • Write functions needed to bootstrap test data — Jan 4th, 2024
    • These should basically be the version that the actual app will use, just deferring all the surrounding functionality that isn't needed to write basic tests
    • Add org
    • Add roles
    • Add user to org with roles/permissions
    • Add new org (using all the above)
    • Add API key for user
  • Password login/logout endpoints — Dec 29th, 2023
  • Get server to start — Dec 27th, 2023
  • Add permissions checks to endpoints — Dec 27th, 2023
  • Add app-specific AuthInfo object — Dec 23rd, 2023
  • Add AuthQueries implementation — Dec 23rd, 2023
  • Select queries need to return the current permission level on each object for project/object-level permissions — Dec 22nd, 2023
  • Add middleware layer to do permissions check — Dec 22nd, 2023
  • Add authz middleware to do generic predicate check — Dec 22nd, 2023
  • query to load user/roles/org for auth info — Dec 21st, 2023
  • Generate router for all model endpoints — Dec 21st, 2023
  • Create endpoints for the model — Dec 20th, 2023
  • Bearer token auth middleware — Dec 20th, 2023
  • Session cookie auth middleware — Dec 20th, 2023
  • Figure out querystring filter parameters — Dec 18th, 2023
  • Pagination — Dec 18th, 2023
  • Rename team to organization
  • Create Rust structures for each model
  • Bootstrap CLI utility
  • Figure out main config format
  • Figure out model definition
  • Create team, user, role, permissions, etc. tables
  • Create SQL migration
  • Create SQL queries to read/write the model

Setup

• This isn't really full documentation., just notes on how I set things up. More of this will be automated and rough edges smoothed out over time.

• Initial Setup

  • Create a git repository
  • Create a Rust project in it
  • Create a SvelteKit project, add filigree-web as a dependency
  • Add the filigree directory with config.toml and other files
  • Create a rustfmt.toml
  • Delete the default src/main.rs
  • Run filigree to write the source code
  • Add this to the vite config

    •   server: {
          proxy: {
            '/api': {
              target: 'http://localhost:7823',
            },
          },
        },
      
      

• Set up Database

  • Create Database and Database User
    • cat /dev/urandom | head | shasum for an easy random password

    • -- Create the user and database
      create role USER login password 'PASSWORD';
      create database THE_DB with owner USER;
      

    • Set DATABASE_URL in your .env
    • cd YOUR_API_DIRECTORY && sqlx migrate run
  • Bootstrap the Database

    • cargo run --release -- db bootstrap \
        --email YOUR_EMAIL \
        --password 'PASSWORD' \
        --name 'YOUR NAME' \
        --org-name 'ORG NAME'
      

    • The only required option is email. Without a password you can log in via OAuth or through a passwordless email login.
    • You can also supply a prehashed password by using the --password-hash argument instead, and hashes can be created by running cargo run --release -- util hash-password YOURPASSWORD.

• Concepts
  • Organization
  • User
    • Users can potentially be in multiple organizations
  • Role
    • Basically a collection of permissions
  • Group
    • This is similar to Role
    • Might not do this, but it can be useful to make this different when more advanced sharing options are created.
  • Actor - all the things (users, roles, groups) that can have permissions on objects
    • When looking at permissions we gather up all the actors that apply to the current user, and use that whole list for permissions checks
• The permissions table is then a list of the permissions each actor has.
• Permissions
  • These should be definable per model once more than one is supported.
  • Tables
    • Global permissions
      • A many-to-many mapping of actor to permission
    • Object permissions
      • A mapping of actor to object and permission
      • For a project-based model, the projects are also just objects
  • Models
    • Role-based Access Control (RBAC)
      • Each role has permissions on whether it can read/write/admin certain types of objects.
      • This is the first one we'll support
      • Might be useful to have two sets of roles, some which can have the global permissions and are only created by administrators, and others which are generic groups of people for managing sharing of projects and objects
        • Need to think about how much this needs to be in the data model vs. just something in the UI. Probably just a single boolean flag is sufficient.
    • Resource-based access control
      • Read/write/admin on individual objects
      • Each role/user has specific permissions for each object
      • More complex, need to make sure the relevant entries get added for each object when it is created
      • Best for a model where objects are not shared by default.
    • Project-based Access Control
      • This combines with the above concepts but is a middle level of granularity compared to the all-or-nothing of plain RBAC and the often-overly-granular resource-based system.
      • A project acts as a container for objects, and actors can have permissions on the project instead of the individual objects.
      • Will probably implement this after RBAC, just need additional functionality for project management and ability to
      • Both individuals and roles can be added to projects
    • Attribute-based Access Control (ABAC)
      • Objects have attributes and actors
    • Public sharing
      • Some sites are built around publicly sharing your stuff.
      • This is basically the same as the resource-based access control model, but with the ability to share with an "anyone" user, and then all the permissions checks check for that actor ID as well.
      • The main change here is that the organization ID checks need to be relaxed when looking for publicly-shared objects.
  • Types of default permissions
    • creator only until shared
    • Creator gets write access, everyone else gets read access
    • everyone can see, only creator can write
    • Everyone in the project can see
    • Everyone in the project can edit

• Types
  • parent-child one to many
  • parent-child one to one
  • Many to one
    • Usually this isn't really a parent-child paradigm so much as related objects. This is actually similar to ActiveRecord belongs_to on the parent object, but without the meaning implied by the word "belongs".
    • e.g. Image uploads can all reference a single conversion configuration, but any image can be switched to a different configuration.
    • The main thing that we want here is the ability to easily fetch the associated objects instead of just their IDs
  • Many to many
    • Almost always best implemented with a join table
• Configuration keywords
  • has - For one-to-one or one-to-many parent-child relationships.
  • has with through for many-to-many relationships
    • In this case through is the name of the model that holds the joins. Making it a full-fledged model allows adding other information to the relationship, if applicable.
    • Need a special mode for through models that will omit the normal id field since its primary key will be a composite of the IDs of the two other models.
  • references for many-to-one relationships
    • This is actually accomplished by the existing references member on a model field definition.
    • Added 2 extra fields that allow setting the option to automatically populate the field on list or get
• Where does the linking field actually go?
  • When there's a linking model, it goes into that model's table, of course.
  • Otherwise for a has relationship it goes into the child object. This is slightly inefficient in a true has_one case but those are rare in practice. In general those turn into either a complex JSON field inside the model itself, or it would be a has_many.

• Add pricing tiers and permissions that go along with those tiers - this should mostly be data driven
  • Need ability for numeric values as well (e.g. max 3 projects, 500MB storage, up to 60 minutes of something, etc.)
• Need ability to do grandfathering of pricing tiers and permissions
  • Essentially this means that both price plans and permissions sets should be versioned, and separately.
  • And there should be the ability to update existing plans and all previous versions

• Workflows are basically state machine definitions which can be instantiated with some data, linked to users/orgs/etc.
• Workflows can perform some set of predefined actions, which may include sending emails, notifying a user, spawning other workflows, etc.
• Workflows can define transitions to run after a certain amount of time
• Consider a scripting engine or something for checking state when transitioning (perhaps just reuse a bunch of code from Ergo for this)

Old Notes

• This should be made modular somehow so it's easy to maintain and add new stuff.
  • Modules define dependencies
• Config
  • formatters to use
  • SQL dialect (postgres or sqlite)
• Model definition
  • Write in TOML
  • Each field has
    • SQL type
    • nullable (default not null)
    • Rust type (can be inferred from the SQL type, but can be overridden)
  • Define Indexes and primary key
    • indexes can maybe just be defined in raw sql
  • Can have child relationships, one -to-one and one-to-many
  • Can have JSON fields, which deserialize to either JSON values or other Rust structs
    • The rust structs can be defined in the config as strings or something like that
  • List of other query functions to generate (find by field, etc.)
  • Permissions
    • Automatic checking on all operations
    • Ability to define which permissions are needed for what
  • some kind of hook on update? or is this better done by just changing the generated code myself
• Consider some kind of codegen for even easier crud endpoint generation
  • Need ability to generate migrations that add/remove columns too
    • this can definitely be manual to start though.
• Authentication
  • Need a user/account mapping to support multiple auth methods per user.
• Scrapped attempt to switch to SeaORM
  • The experience starting to try this ended up with Rework query template system — Jul 16th, 2024
  • Problems
    • exec_with_returning only supports returning a single model, this isn't an issue with raw sqlx
    • find_with_related only supports a single related model, not an issue with the current query building
  • But the nice things were...
    • Easy to add auth query with a wrapper
    • Easy dynamic query construction
    • ActiveModel is pretty nice
    • Easier to decide at runtime which things to populate
    • Templates were easier to deal with, no more gnarly templates with a bunch of ifs generating SQL or functions to call queries.
    • Partial Models were nice

• Usually want PSM mode 4 (single column of text) or 6 (single uniform block of text)
• Tesseract Page Segmentation Modes (PSMs) Explained: How to Improve Your OCR Accuracy - PyImageSearch

• # get grayscale image
  def get_grayscale(image):
      return cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
  
  def thresholding(image):
      return cv2.threshold(image, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]
  

• Deskew

• Multi-Column Table OCR - PyImageSearch
• Microsoft Table Transformer (TATR)

• Some videos really should have just been blog posts, and I feel like I've just wasted time.
• Others are actually valuable, but as a parent with young kids my time is limited, and so this allows faster consumption or in settings where watching a video isn't feasible.

Task List

• Up Next

  • Host this somewhere so I can access it over the web and add to Readwise
    • Maybe just host locally and expose subdomain

• Soon

  • Get thumbnail image from video metadata
  • Summarize comments (can yt-dlp grab them?)
  • For videos with chapters, summarize each chapter
  • Table of contents with video chapters
  • Improve reader layout

• Later

  • For longer videos without chapters, use LLM to try to generate chunks
  • Click on/near a paragraph to play the video at that point.
  • Better image similarity algorithm
  • Some way for multiple users to link to the same video without downloading it each time (if opened to public)
    • Probably put the video metadata itself in a separate table and have the user's video model link to that.
  • Video Host whitelist? (if opened to public)
    • Might be a hassle to maintain but also potentially prevents issues with shady content

• Done

  • Move processing into Rust
    • Steps
      • Download video
      • Images
        • Extract Images
        • Similar Image Algorithm
      • Transcript
        • Extract Audio
        • Transcription
        • Summary of Transcription
  • Simple auth
  • Allow storing files in cloud storage
  • Use Deepgram for transcription
  • Hosting: Store non-temporary downloaded files in Backblaze or Cloudflare object storage
  • Integrate with Filigree
  • Allow triggering video download and processing from inside the app — Nov 20th, 2023
  • Queue up multiple videos for processing — Nov 20th, 2023
  • Add "read" state and progress tracking — Nov 20th, 2023
  • Speed up SSIM process — Nov 18th, 2023
  • Generate a summary of the video — Nov 18th, 2023
  • Use structural similarity (SSIM) metric to remove duplicate screenshots — Nov 17th, 2023
  • Screenshot extraction
  • Whisper transcript generation
  • Align transcript to images and

<script lang="ts">
  import Papa from 'papaparse';
  
  let columns: string|undefined;
  let previewRows: Array<Record<string, string>> = [];
  
  async function openFile(file: File|undefined) {
    if(!file) {
      return;
    }
    
    Papa.parse(file, {
      header: true,
      preview: 3, // To read first 3 data rows
      skipEmptyLines: 'greedy', // Well-formed CSV files don't need this
      complete(parsed) {
        if(parsed.errors?.[0]) {
          error = parsed.errors[0].message;
        } else if(parsed.meta.fields) {
          columns = parsed.meta.fields;
          previewRows = parsed.data;
        }
      }
    })
  }
</script>

<input 
  type="file"
  name="csvfile"
  accept=".csv,text/csv"
  on:change={(e) => openFile(e.currentTarget.files?.[0])} 
/>

• description = "Summarize some files"
  
  # This can also be template_path to read from another file.
  template = '''
  Create a {{style}} summary of the below files
  which are on the topic of {{topic}}. The summary should be about {{ len }} sentences long.
  
  {% for f in file -%}
  File {{ f.filename }}:
  {{ f.contents }}
  
  
  {%- endfor %}
  '''
  
  [model]
  # These model options can also be defined in a config file to apply to the whole directory of templates.
  model = "gpt-3.5-turbo"
  temperature = 0.7
  # Also supports top_p, frequency_penalty, presence_penalty, stop, and max_tokens
  
  [options]
  len = { type = "int", description = "The length of the summary", default = 4 }
  topic = { type = "string", description = "The topic of the summary" }
  style = { type = "string", default = "concise" }
  file = { type = "file", array = true, description = "The files to summarize" }
  

Task List

• Up Next

  • Switch request layer to use Chronicle

• Soon

  • Verbose mode should print token stats at end
  • List command
    • List all templates in a directory
    • should also take a filter
    • Short mode for completion
    • by default print template name and description in a table
  • Show command to output the information from a template
  • "run" command detection is fragile
  • Add tools definitions to templates
    • This lets us run a script and get a JSON output

• Later/Maybe

  • Save all invocations in a database? (will do as part of Chronicle switch)
  • Allow templates to reference partials in same directory
  • Allow templates to reference partials in parent template directories
  • Define ChatGPT functions in the prompt? Probably skip this, more appropriate for some other project
  • bash/zsh Autocomplete template names
  • Can we autocomplete options as well once the template name is present?
  • Recall previous invocations
  • Option to trim context in the middle with a <truncated> message or something like that

• Done

  • Pass images to Ollama — v0.3.0 Dec 13th, 2023
  • Support for GPT4 Vision — v0.3.0 Dec 13th, 2023
    • Can pass images in base64
  • Support OpenRouter — v0.2.0 Dec 8th, 2023
    • OpenRouter offers an OpenAI compatible API which is probably the easiest way to add this.
  • Set prompt format, context length, etc. per model - v0.2.0 Dec 8th, 2023
    • Done specifically for Together right now, can expand this to generic host at some point
    • Support standard formats and allow custom formats too
    • Needed for some providers that don't apply the template for you, or who don't provide accurate info about context length and other things.
    • This can be defined in the model definition once it can be an object (see Support multiple hosts - v0.2.0 Dec 8th, 2023).
  • Support together.xyz model host - v0.2.0 Dec 8th, 2023
    • Fetch model info from https://api.together.xyz/models/info
    • Short term cache on the model info
    • Get the config from the model info to determine how to format the prompt, stop tokens, etc.
      • Some of the configs here actually don't include things like system prompt...
        • Maybe just build in templates to the tool and lets them be specified in the config somehow
      • Looks like context length is missing from some models as well
    • max_tokens seems have a very small default, need to set this higher to be useful
  • Support multiple hosts - v0.2.0 Dec 8th, 2023
    • Allow defining hosts beyond the built-in hosts
      • API url
      • Request Format (currently just OpenAI and Ollama format but probably more in the future)
      • Environment variable that holds the API key
    • Ability to configure the default host for non-GPT-3.5/4 models (whereas now Ollama is the default)
    • Need a way to specify in the model name which host to use
      • Actually the way to do this is to allow the model name to be either a string or a { host: Option<String>, model: String } structure.
    • Tests
      • config file overrides specific fields of bulit-in hosts
        • e.g. host.openai.api_key = "DIFFERENT_VAR_NAME"
      • Adding new hosts from the config file
      • Use default provider when none is specified
      • Set default_host to something else
      • Complain when default_host refers to a nonexistent host
      • Alias handling
        • Alias can be a full model spec
        • Model can be a full model spec and also reference an alias which is a full model spec. Should fetch the alias from model and merge the remaining fields together in this case
  • Testing
    • stop at the top_level config
    • Resolution of model options between different configs
    • Don't require a config in every directory
    • Malformed configs raise an error
    • Malformed templates throw an error
    • templates resolved in order from the current directory
    • Look under ./promptbox.toml and ./promptbox/promptbox.toml
    • Prompts can be in subdirectories
    • Prepend
    • Append
    • Prepend and append
    • all types of arguments
    • Bool arguments are always optional
    • required arguments (switch from required to optional)
    • Array arguments
    • Template model options should override config model options
    • make sure it works to invoke with command-line options and template options at the same time
    • system prompt, embedded and in separate file
    • json mode
  • Handle 429 from OpenAI — v0.1.2 Dec 4th, 2023
  • Chop off too-large context, option to keep beginning or end — v0.1.1 Dec 1st, 2023
    • Should also be able to specify which inputs to slice off i.e. keep the fixed template intact but remove some of the piped-in input
    • Ideally have per-model context values.
      • Ollama can get this from the API.
      • OpenAI has few enough models that we can do some basic pattern matching to make a guess
      • But need ability to specify a lower cap too, e.g. maybe we never actually want to send 128K tokens to GPT4
  • Token counter functionality — v0.1.1 Nov 30th, 2023
  • Set up CI and distribution — Nov 21th, 2023
  • Streaming support for openai — Nov 14th, 2023
  • Append any additional positional arguments
  • Append input from stdin
  • Support format="json"
  • Streaming support for ollama
  • Integrate with ollama
  • Option type to paste a file contents in, and allow wildcards for array files
  • Send request to model
  • Move the main command to be a "run" subcommand
  • Basic functionality
  • Define CLI options in template file
  • Help output always shows openai_key (maybe due to .env?)

General Tasks

• Set up a justfile to run common tasks

  • _default:
      @just --list
    

JS Projects

• husky to install Git hooks
• For public projects, set up Github PR template with a note to add a changeset.
• lint-staged to help with running prettier on commit
  • Sample .lintstagedrc file
• changesets for changelog generation
  • Add @changesets/changelog-github for including Github PR info in changelogs
  • Add these package.json scripts

    • "changeset": "changeset",
      "generate-changelog": "source ghtoken; GITHUB_TOKEN=${GITHUB_TOKEN} changeset version",
      

  • Install changeset-bot.

• ESLint

  • Disable some annoying rules which are enabled by common config packs

  • rules: {
        'prefer-const': 'off',
        '@typescript-eslint/explicit-module-boundary-types': 'off',
        '@typescript-eslint/ban-ts-comment': 'off',
        '@typescript-eslint/ban-types': 'off',
        '@typescript-eslint/no-empty-function': 'off',
        '@typescript-eslint/no-non-null-assertion': 'off',
        '@typescript-eslint/no-unused-vars': 'warn',
     }
    

• Prettier

  • Install prettier-plugin-tailwindcss

  • {
      "useTabs": false,
      "singleQuote": true,
      "trailingComma": "es5",
      "printWidth": 100,
      "plugins": ["prettier-plugin-svelte", "prettier-plugin-tailwindcss"],
      "overrides": [{ "files": "*.svelte", "options": { "parser": "svelte" } }]
    }
    

  • For now, update package.json format and lint scripts to workaround a bug in Prettier 3.0. (Note: this seems to be fixed, but need to confirm)

    •     "lint": "prettier --ignore-unknown --check './**/*' && eslint .",
          "format": "prettier --ignore-unknown --write './**/*'"
      

Rust Projects

• Set up rustfmt.toml the way I like it

  • edition = "2021"
    imports_granularity = "Crate"
    group_imports = "StdExternalCrate"
    

Task List

• Up Next

  • Option to store apps within database
    • Each app's code has two Javascript functions, one which does the work and another which formats the result
    • These apps should be bundled and then run on the backend.
    • Add a new table: app_code, which is linked to the app id, contains the code, and has a version r
      • Hold both the original code and the bundled version.
    • An app with in-app code links to a specific row with the same app id
    • When running, execute using JS Sidecar
  • In-app code editor
    • Can take a lot from Ergo for this but worth seeing if there's new stuff for the code editor
      • https://github.com/val-town/codemirror-ts for example might have some improvements over what I wrote before, need to check
      • What's the best way to do bundling now? The existing method worked pretty well, probably worth just continuing that.
    • Code should have a "latest" version which autosyncs and then the active version activates when clicking save

• Soon

  • In App Code
    • KV interface for functions to persist data
      • Consider just implementing a light version of node:fs which uses SQLite behind the scenes
        • Won't really work with seeking or anything but that's ok, doesn't come up often anyway
    • AI code generation
    • AI-generated UI
  • Inbox-style interface
  • Notifications
  • Customize layout of the data items
    • Get rid of the existing format for items, and make this into a block-based thing with rows and columns, where blocks can be text/image/chart/etc.
    • Block types
      • Text
      • Chart
      • Row
      • Column
      • Expansion Pane
      • Image
    • Blocks can reference a separate data block using JSON pointer syntax
    • Text can be markdown
    • Probably keep "title" and maybe subtitle but the rest should be new

• Later

  • Calculate a hash of all data fields and store/compare in the database to avoid unneeded writes when nothing has changed. Can possibly use the new MERGE syntax to help with this.
  • Publish live on web
  • Menubar App
  • Item actions
    • See Apps can define actions to be done.
  • Detail pages can be a web app served by the app itself?
  • Main Item display can be a simple chart or other viz
  • Item detail can be a vega viz or something
    • See Details
  • Let apps expose their config through the platform
    • Things like your location for getting the weather
  • SSE update notification

• Done

  • REST endpoints to add/update/delete an app — Jul 29th, 2024
  • REST endpoints to add/update items within an app — Jul 29th, 2024
  • Simple auth system so that I can publish this readonly but still log in and make changes — Feb 14th, 2024
  • Platform can run apps on a schedule — Dec 12th, 2023
  • Web app reloads data on a schedule — Dec 11th, 2023
  • Dismissible items — Nov 28th, 2023
  • Render simple UI — Nov 28th, 2023
  • Figure out more structured layout for app data — Nov 27th, 2023
  • HTTP Server — Nov 27th, 2023
  • Read items from each data file
  • Watch app data directory
  • Write a simple app that always has data to show (Weather or HN API or something super basic)
  • Create app helper library
    • Common paths, JSON data definition, etc.
  • Firm up designs
  • Design JSON schema for data

Basic Design

• The platform has a directory, where each app can place a JSON data file.

• App Data Format

  • This is a JSON file with some information about the app and its current data.
  • App Title
  • URL to reverse proxy to for detailed web UI
    • And index page path, if applicable
  • Data Items Array
    • Something like this
    • Item UUID
    • Item type
    • Description
      • This is HTML that gets injected into the page.
    • Chart?
    • Structured data
      • This can be used to allow more customization of the displayed data
    • Notification - optional
      • Notification also has a UUID so that a single item can show multiple notifications even if it already exists.
      • Notification title and text
      • Icon/image?
    • Persistent or dismissable
      • Dismissable items can be closed by the user until they are updated again, if ever.
    • Updated time
    • Details
      • This can be a path under the reverse proxy URL, if present
      • Or some defined visualization
        • This will be something like a SQLite database, a set of queries to run, and maybe some Vega viz to render?

• Backend Platform

  • Watch the data directory
  • On load and when a file updates, read the file and see what the data items are
  • Serve a web app which shows all the data
  • Track things like notifications, dismissible item state, and so on.

• UI

  • Menubar app
  • More detailed page for use in a browser/phone.
  • Good case for some kind of microcontroller with an OLED
  • Should be possible to run all these on a different computer from the platform.

Apps

• Weather
• Hurricane and weather alert status
• HN and Lobste.rs top links with actions for summarization
• Watch Github releases for certain projects

Notes

• V1 of this is a few components:
  • A mini-app template project that can be forked
  • A main server
    • Manage and start the mini-apps
    • Set environment for the mini-apps with some well-known keys:
      • directory for each app to create its SQLite database and other stuff like that
    • Provide a way the mini-apps can call to push new data
      • indicate that a notification should be shown for important stuff
      • push new data to be shown when the app is opened
      • some way to define how to lay out the UI for that data
      • Could be HTTP endpoints that the server exposes
      • Items can be persistent until the app remove them, or acknowledgeable, where the stay until the user dismisses them.
      • Actually just using the filesystem would work well
        • Data goes in a sqlite database
        • schema file defines both visualizations and the queries to run to create the data for them.
        • Server watches for changes in the schema file.
        • How do we read all the data in an efficient manner?
          • Having to read the dashboard databases all at once when the dashboard opens could be messy.
          • OTOH it could be massively parallelized and the reads would generally be very quick, just a few rows per DB.
          • In reality, having all the summary data in the JSON itself will be fine. There won't be that much data.
        • Notifications would probably still need an endpoint?
          • Could have a field in the JSON that would let an app set a notification, with a UUID or something to handle idempotency.
    • A web-based UI
      • Maybe both as a menu-bar app and as a normal app
      • Lists and manages the mini-apps
      • Displays the pushed data
• V2
  • Some way for mini-apps to expose a more in-depth look into their data.
    • Probably do this through some kind of reverse proxy that just assigns some path to proxy through to the mini-app so it can serve its own UI.
  • Apps can define actions to be done.
    • Each item can have its own actions
    • Actions have a label, maybe an icon, styling, and a way to do the action
      • HTTP endpoint to hit.
      • Website to visit for more details
      • Command to run
  • Auto-deploy from Git/HTTP sources?
  • LLM summarization of current stuff?
  • AI Agents which can send their status and results to the app (this probably wouldn't require any new functionality)
• How to run mini-apps?
  • In some ways, it would be simplest for the app to be a single file which gets started by a worker. The file can export certain functions to deliver the relevant functionality.
  • A full-fledged app is much more flexible, but also somewhat harder to distribute and harder to proxy to. I think in the end this will be a better solution though.
    • Distribution: this can be a zip file. Run bun to install deps and then start it. This also technically allows the server to be written in any language, though that adds complexity.
  • Could just have them run separately, which would be simplest in some ways. Some apps would run persistently, but many apps would just be a cron job. The cron jobs could also be done as an internal system in the platform.

Notes from Initial Idea

• Perhaps what I really want is just a way to have some kind of PaaS system for mini-apps, where each one is somewhat separate despite being hosted behind a single place. So this turns into a reverse proxy with easy-to-add upstreams at arbitrary paths, and an easy way to write servers and set up storage, manage notifications, etc.
• So we have a server which handles the reverse proxy. Each mini app is hosted at a separate path on the server but otherwise runs separately. Notifications and other actions are also endpoints in a private part of the server which the mini app can call back into. Maybe function as a tracing sidecar too?
• How much of this could be done with off the shelf components?
  • Consul for service registration
  • Vault for pulling secrets like webhook keys
  • Nomad for running the mini apps and pulling in the secrets into the containers
  • caddy or nginx for the web server, using dynamically generated templates from consul service records?
  • Task scheduler to poke services every so often, or should this be built in to the apps themselves?
    • How do we manage concurrency here? Probably by just having each app that needs it manage its own queue. Could do this using effectum to make it easy.
    • Would it be useful to provide a shell library that could allow writing each service as just a simple function or as close as possible? I think this would be helpful. Create the database connections, pull in the proper secrets, and so on. Apps that needs more functionality could use an http router or SvelteKit or something. But a simple approach like this could potentially allow for creating mini-apps within some other UI again, perhaps even with dashboard building blocks.
• Is there a good platform in here to experiment with AI coding solutions as well?
• I think I'm ending up with 2 models here:
  • mini apps that are actually full apps
  • Mini apps that are actually just executable functions but with some extra functionality compared to what's there now in Ergo.
  • So the former is just a matter of hosting, proxy, etc. The latter is where the design comes in.
• Micro apps (probably not going to do this)
  • single function to execute the actual code
  • Optional schedule on which the function runs.
  • UI defined with simple building blocks
    • Queries to run against the stored data, something like a stored procedure
    • Charts to display those queries
    • controls to update the parameters for the queries
    • A way to define all this easily, not necessarily no-code, but low-code to start, similar to Gradio
  • compared to the current method, the function should be uploaded from a file and executed in something external like Node, Deno, or Bun.
    • This allows for it to actually be tested and developed externally, without much fuss.
    • Optionally, perhaps even update these functions from a Git repository, where pushing to the master branch updates the function to be run.
    • Then we could just do a shebang to decide in what context it actually runs (with the inherent security problems this has? Maybe a whitelist of safe executors, and this is especially appealing with Deno and Bun(?) which have permissions systems).
    • Also look into WASM execution now that WASIX and similar standards are making that easier.
    • Provide an SQLite database path for each function so that it can implement its own persistence.
  • An app should provide four functions: query, act, cron (optional) and buildUI.
    • Query gets data from the database for the UI to show
    • Act performs actions based on button clicks
    • Cron, if implemented, will do scheduled data updates
    • buildUI shows what kind of dashboards to show
    • Something like this anyway. Need to figure out the relation between query and buildUI, maybe they should be merged.
  • Application package defines:
  • sqlite database connection with predetermined location
  • Sqlite migration system
  • A way to define UI via code
  • A way to send notifications through the app host
  • a job queue
• Or... just make templates, use the template, and have a way to autodeploy them all behind an reverse proxy.
  • The main downside of this is that it's harder to make a dashboard with status from various apps all in one.
  • But this could be done through a common endpoint that returns data and ui definitions.
  • So in this paradigm we have:
    • server that exists independently and with its own UI
    • a separate dashboards app that apps can register with
    • A /dashboard endpoint on each server that tells the dashboard what to display.
    • Or an endpoint on the dashboards app through which the servers can push data. This is probably the way to go since it doesn't enforce any particular structure for each app.

Extracting Contacts

• The address book file is in the backup at 31/31bb7ba8914766d4ba40d6dfb6113c8b614be442. This is a SQLite3 database.
• The main tables you care about here are ABPerson and ABMultiValue. To do a simple extraction of names and phone/email you can use a query like this.

  • select ABPerson.last, ABPerson.first, ABMultiValue.value
    from ABPerson,ABMultiValue
    where ABMultiValue.record_id=ABPerson.ROWID order by Last, First
    

Locating Files

• The Manifest.db is a database of all the files. You can run a query like this to find a file.

  • SELECT fileID, relativePath
    FROM Files
    WHERE relativePath like '%Address%';
    

• From there, the first two digits of the fileId indicate the directory to look into, and the entire fieId is then the filename in that directory.
• The pypi page for the iOSbackup Python package also has a list of commonly needed files and their fileID values.

Task List

• Up Next

• Soon

  • Basic intent detection
  • Run web searches and generate an answer from the results.
  • read system message from a file
  • record llm pipeline actions for later analysis

• Later

  • Check out whisper-turbo for in-browser voice recognition
  • optional configuration to use better models that require GPU

• Done

  • set up basic ChatGPT workflow
  • Voice recognition
  • Basic TTS
  • Websocket-based communication
  • Stream results back to client

System Prompt Example

• You are Buzzy, an AI bot that answers questions for children. Your answers should be appropriate for a smart six year old boy, but also don't dumb your answers down too much.

Ideas

• Do I want to do a RAG-based conversation learning/memory?
• Decide whether to send back previous chat messages based on how much time has passed.
• Voice recognition to ask questions
  • Choice:
    • Decision: nvidia/stt_en_fastconformer_transducer_large
    • Options:
      • Run Whisper in browser?
      • Whisper on server?
        • Too slow when running on CPU
      • nvidia/stt_en_fastconformer_transducer_large
        • runs fast (~500ms for short passages) and works well
  • Needs to run ok on just CPU
  • Seems to work best to just send the audio in one big chunk to the server.
  • Huggingface Voice Recognition Leaderboard
• TTS to say responses
  • How easy is it to use one of the new models for this?
  • Choice:
    • Decision: Mimic 3
    • Options:
      • Bark seems promising
        • https://github.com/suno-ai/bark
        • https://github.com/serp-ai/bark-with-voice-clone
        • https://github.com/gitmylo/bark-voice-cloning-HuBERT-quantizer
        • Very slow on CPU, RTF of 6-8
      • XTTS
        • https://github.com/coqui-ai/TTS/blob/dev/docs/source/models/xtts.md
        • Sounds the best, but has RTF of 4-5 on CPU
      • Mimic 3
        • This turns out to be the only solution that both sounds good and runs in realtime on CPU
        • Running with lengthScale 1.2 slows it down a bit and seems to give best results
• Some kind of 3D avatar that's like a dinosaur or robot or something?
• Scrensaver mode that does a photo carousel

Intent Detection

• DeBERTa-v3-base-mnli-fever-anli seems to work well for this at first try. Haven't really exercised it significantly yet though. The creator of that model now also has deberta-v3-large-zeroshot-v1.
• Tasks
  • Figure out if something is a question that can be answered by searching the web and/or wikipedia
  • How many days until...
  • Show me pictures of...
• When doing web search and intent , also need to detect if a query builds on the previous queries or not.
  • "No, the blue one" has no context but the context is probably in the previous message.
  • Small models don't seem to do great on this but gpt-3.5-turbo-instruct does well.

    • Assistant: {assistant's last message}
      
      User: {user's question}
      
      Does the user's question ask for clarification on the assistant's statement? Only answer yes or no.
      

  • For this we can probably just pair up the last assistant message with the latest query, since they tend to include all the necessary info again in every message. Then use GPT to create a proper search for it.
  • Do we even need to do the detection? Will it work to just ask GPT to make a search for the query?
    • Takes some tweaking of the prompt but this seems to work well.

    • This is an excerpt of a chat with an assistant and a user:
      
      Assistant: {assistant's last message}
      
      User: {user's question}
      
      What would be a good web search to answer the user's question? If the question is asking for clarification on the assistant's statement, then the web search should account for that. If it is a new line of questioning, then ignore the assistant's statement. Respond only with the web search and nothing else.
      
      Web search:
      

Web Search

• Use Brave Search API to do web searches to answer questions
  • Should searches be an intent, or should we run searches for anything that doesn't return another intent?
  • Maybe also wikipedia/wikidata?

Task List

• Up Next

• Soon

  • Global queue system and persistent workers for tasks
    • This should speed up the time between tasks a lot and makes things more flexible as new tasks are added dynamically.
    • Start a pool of workers, each of which is qualified to handle one or more stages
    • Each worker communicates with the manager and fetches tasks from it
    • The biggest change this requires is ensuring that the manager and the workers can actually communicate. Can be an issue when it requires punching a hole in a VPC for example, maybe there's a better way to do it.
      • Just running the manager as a task inside the VPC as well would accomplish some of this.
      • Opening a hole in the VPC could work:
        • Set up VPC ACLs to allow inbound comms on some port
        • Manager generates a client SSL cert and sends the pubkey along with the worker payload
        • Worker gets a public IPv6 IP
        • Once a worker starts,
          • manager connects to the worker
          • worker verifies the client cert
          • uses a 2-way GRPC connection to swap info both ways
          • This would also help with shipping logs back to the manager.
      • Could also work to use some other queue service like SQS which is easy to talk to from both sides
        • But this requires supporting a bunch of different APIs on both sides.
    • This also opens things up in some ways for running in Spot mode
      • Add a function that allows a worker to checkpoint its work. The checkpoint function sends a payload to the manager, and then if the worker dies during the execution the manager can put the task back in the queue with checkpoint info, and the worker can resume from where it left off, if possible.

• Later/Maybe

  • Platform adapters could make it easier to build the local and platform-side code
    • Not sure how useful this actually is, but this would mostly be a preconfigured way for a binary to determine if its running in the platform or not and then run your manager code or your worker code when you're doing the single-source-code configuration
    • I think a simple abstraction would be something like:
      • The spawner has an easy way to set a "SMELTER_TASK" variable in the environment (or whatever is the equivalent) of the spawned task.
      • Provide a function that retrieves this value, and then you can match on it and run your own code. If it's not set, then you assume that you are the job manager and run that code instead.
      • Ultimately this isn't too different from doing it yourself, might be a nice convenience though especially when developing the pipeline.
  • Ability for spawner to send an event when a container goes from pending to running
  • Option to run a server within the worker container that can collect logs and stream them back to the manager
  • Option to timeout if we don't see any logs for too long
    • This depends on being able to get logs shipped from the container down to the manager.
  • AWS Lambda adapter
  • Support Fargate spot instances
    • A way for SpawnedTask to indicate that a task failure is from a spot instance shutdown
      • This can be a new error code alongside TaskFailed, call it TaskPreempted
    • Specify which capacity provider to use for Spot vs. Normal launching
    • An option to launch it as a non-spot instance if the spot instance gets preempted too much or if it takes too long to start.
    • Currently Fargate doesn't support spot on ARM
  • Convert a single SQL statement into a multi-stage set of tasks
    • This may be better done by integrating with something like Datafusion which already supports creating query plans
      • Check out https://github.com/datafusion-contrib/ray-sql and https://github.com/datafusion-contrib/datafusion-federation which have some common concerns.
    • Consider also supporting Substrait query plan format
      • DuckDB supports consuming Substrait so I may be able to generate Substrait from SQL, split it into stages, and then run the nodes as needed
  • Clean up inter-level channels
    • There are a bunch of channels for communicating cancel state, task results, etc. between the tasks, stages, job, and manager. See if there's a way to make this cleaner.

• Done

  • Retry inside the spawner when Fargate returns a capacity error
  • Cancel all the jobs when an error occurs, and wait for them to be cancelled
  • Handle rate exceeded errors from Fargate API
    • The Rust AWS SDK doesn't properly handle ThrottlingError and flag it for retry, but with a custom retry layer this can be done.
  • Workers automatically track RAM usage, load factor, time running, etc., and return these with the task results
  • Full test of Fargate jobs
  • Allow a cancel signal to kill all containers
  • StatusSender needs to work with a raw StatusUpdateItem, and the StatusCollector should be able to handle native StatusUpdateItem and have its own separate mechanism for the read operations. (Maybe just use a mutex here to simplify?)
  • Build fargate container
  • Have something that helps generate task definitions for fargate
  • AWS Fargate Spawner
    • How to figure out if a container succeeded or not? — exit code is exposed though there's no other way to pass back data
  • AWS Fargate Worker Framework
  • Example to run workers as local processes
  • Remove the Spawner trait altogether
    • It's no longer necessary and isn't sufficiently expressive for most spawners

Test Datasets

• https://opensky-network.org/datasets/raw/

Initial Features

• Split up queries across a number of workers
  • Split queries into chunks
  • Max number of concurrent workers
• Run workers
• Gather results
• Reduce results using 1 or more reducers
• Retryable workers with some threshold for retrying
  • e.g. autoretry remaining workers once the first 90% have returned

Pluggable Backends

• Execution
  • AWS Lambda
  • Docker/Kubernetes/Nomad
• Storage
  • Various Cloud storage

Applications

• Query over some data and output a DuckDB file with the results for further analysis.

From Honeycomb's O11y Book But serverless functions and cloud object storage aren't 100% reliable. In practice, the latency at the tails of the distribution of invocation time can be significant orders of magnitude higher than the median time. That last 5% to 10% of results may take tens of seconds to return, or may never complete. In the Retriever implementation, we use impatience to return results in a timely manner. Once 90% of the requests to process segments have completed, the remaining 10% are re-requested, without canceling the still-pending requests. The parallel attempts race each other, with whichever returns first being used to populate the query result. Even if it is 10% more expensive to always retry the slowest 10% of subqueries, a different read attempt against the cloud provider's backend will likely perform faster than a "stuck" query blocked on S3, or network I/O that may never finish before it times out. What happens if someone attempts to group results by a high-cardinality field? How can you still return accurate values without running out of memory? The simplest solution is to fan out the reduce step by assigning reduce workers to handle only a proportional subset of the possible groups. For instance, you could follow the pattern Chord does by creating a hash of the group and looking up the hash correspondence in a ring covering the keyspace.

Pipelines

• from transformers import pipeline
  
  classifier = pipeline("sentiment-analysis")
  
  

• Full list of premade pipelines: https://huggingface.co/docs/transformers/main_classes/pipelines
• Pipelines take a model argument. Some models also specify a particular pipeline to use, and in that case you can omit the pipeline name.

• generator = pipeline("text-generation", model="distilgpt2")
  

• Mask model for filling in words

• from transformers import pipeline
  
  unmasker = pipeline("fill-mask")
  unmasker("This course will teach you all about <mask> models.", top_k=2)
  
  

• Pipeline Implementation

• Each pipeline is just running the few steps needed to run a model. For example, a BERT sequence classification pipeline may do something like this:

  • from transformers import AutoTokenizer, AutoModelForSequenceClassification
    import torch
    
    text_inputs = [
        "I've been waiting for a HuggingFace course my whole life.",
        "I hate this so much!",
    ]
    checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
    
    # Tokenize the input
    tokenizer = AutoTokenizer.from_pretrained(checkpoint)
    inputs = tokenizer(text_inputs, padding=True, truncation=True, return_tensors="pt")
    
    # Run the model
    model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
    outputs = model(**inputs)
    
    # Softmax to convert from logits to probabilities
    predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)
    # predictions == tensor([[4.0195e-02, 9.5980e-01],
    #        [9.9946e-01, 5.4418e-04]], grad_fn=<SoftmaxBackward>)
    
    model.config.id2label
    # {0: 'NEGATIVE', 1: 'POSITIVE'}
    # So the first input is positive, second input is negative.
    

Specific Model Types

• In addition to AutoModel the transformers library provides classes for specific model types, such as BertConfig and BertModel. In most inference-only cases AutoModel is fine though.
• When not training a model from scratch, you will usually preload a specific config.

• model = BertModel.from_pretrained("bert-base-cased")
  tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
  

• from_pretrained will also download the weights and related files, if needed.
• When you have performed additional training, you can use model.save_pretrained(filename) to save the config and the weights to disk.

Tokenizers

• Tokenizers can both go from raw text to token IDs with tokenizer.tokenize and tokenizer.convert_tokens_to_ids, and back from token IDs to words again with tokenizer.decode, which will both convert IDs to tokens and combine subword tokens into full words.
• The padding token ID can be retrieved from tokenizer.pad_token_id.
• Attention masks can be used to tell the model to ignore certain tokens. This usually matches to the locations of the padding tokens being 0 and everything else being 1.

• ids = torch.tensor([...])
  attention_mask = torch.tensor([...])
  
  outputs = model(ids, attention_mask=attention_mask)
  

• Depending on the tensor framework in use, you can ask the tokenizer for different types of tensors.

• inputs = tokenizer(texts, return_tensor="pt") # PyTorch
  inputs = tokenizer(texts, return_tensor="tf") # TensorFlow
  inputs = tokenizer(texts, return_tensor="np") # NumPy
  

• Tokenizers support all the standard configuration
  • truncation=True to truncate inputs longer than model context. max_length=16 to use a custom truncation length
  • padding=True to pad inputs to the same length

Training

• To train a model, you tokenize your inputs and then add an additional labels property which is a tensor with the expected answer for each one.

• import torch
  from transformers import AdamW, AutoTokenizer, AutoModelForSequenceClassification
  
  checkpoint = "bert-base-uncased"
  tokenizer = AutoTokenizer.from_pretrained(checkpoint)
  model = AutoModelForSequenceClassification(checkpoint)
  
  sequences = ["string1", "string2", "etc"]
  batch = tokenizer(sequences, padding=True, truncation=True, return_tensors="pt")
  
  # The answers for each of the above
  batch["labels"] = torch.tensor([1, 0, 1])
  
  # Single step, usually a whole training loop would go here.
  optimizer = AdamW(model.parameters())
  loss = model(**batch).loss
  loss.backward()
  optimizer.step()
  

• Because bert-base-uncased is not originally set up for sequence classification, the library will discard the original model head and add a new randomly-weighted head for sequence classification.
• The datasets library automatically splits a dataset into training, validation, and test sets.
• dataset.features describes the feature names and types, including (when applicable) the descriptions of what each label number actually means
• You can use dataset.map to tokenize while keeping all the data in the much more efficient Apache Arrow format. It also does multiprocessing and caches results.
  • e.g. for a BERT next-sentence prediction:

  • from transformers import DataCollatorWithPadding
    
    def tokenize(row):
      return tokenizer(row["sentence1"], row["sentence2"], truncation=True)
    
    tokenized = dataset.map(tokenize, batched=True)
    
    # Using the collator to pad this way per batch is more efficient than padding everything to the max length across all items
    collator = DataCollatorWithPadding(tokenizer=tokenizer)
    batch_size = 512
    samples = collator(tokenized["train"][:batch_size])
    

• With that set up, you can start your training loop using the Trainer class, which handles all the batching, gradient descent, etc.

• from transformers import TrainingArguments, Trainer, AutoModelForSequenceClassification
  import evaluate
  # Can also pass `push_to_hub=True` to automatically push to Huggingface Hub when done
  training_args = TrainingArguments("directory-to-save-to", evaluation_strategy="epoch")
  
  model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
  
  # A function to report metrics at the end of each `evaluation_strategy` from the TrainingArguments
  def compute_metrics(eval_preds):
  	metric = evaluate.load(same arguments that loaded the dataset)
      logits, labels = eval_preds
      predictions = np.argmax(logits, axis=-1)
      return metric.compute(predictions=predictions, references=labels)
  
  trainer = Trainer(
  	model,
  	training_args,
  	train_dataset=tokenized["train"],
  	eval_dataset=tokenized["validation"],
    	# This can be skipped if using a DataCollatorWithPadding since that's the default when omitted.
  	data_collator=collator,
  	tokenizer=tokenizer,
    	compute_metrics=compute_metrics
  )
  
  trainer.train()
  

• Full training loop example at https://huggingface.co/learn/nlp-course/chapter3/4?fw=pt
• Once the model has finished training, the Trainer will let you run the model.

• predictions = trainer.predict(tokenized["validation"])
  # { predictions: [predicted logits for each row], label_ids: [correct answers], metrics }