BrainExpanded – Web app and Data Sources

As I wrote in previous posts, the manual recording of memories for BrainExpanded is just the beginning. Once the basic infrastructure is in place, devices (e.g. phones, laptops, wearables) and applications/services can automatic capture moments from a user’s life such as the articles they encounter, the notes they take, the activities throughout their lives, the places they visit, the people with whom they interact, the discussion they have, and so much more. BrainExpanded may also connect to existing data sources such as email, calendar, and messaging.

Sam Goodwin, an ex-colleague from my Amazon days, pinged me to bring the following post by Ian Nuttall to my attention, which was liked by a good number of people.

Well, this should be easy for BrainExpanded so I spent a couple of days evolving the system to support it.

The System Design is evolving

In addition to the iOS app, I wanted to introduce a client experience that can run in the browser on top of the same Web API. So there is now a new layer in the architecture which is built using React to support the BrainExpanded Web app.

I had already implemented authentication using Google’s Firebase SDK for the iOS app so I did the same for the Web app. Dealing with CORS and JWT-based authorization in ASP.NET Core for the Web API was a bit tricky but now everything is in place. All the Web APIs are now properly protected; they require a Google ID token which is validated for every request.

I have also introduced the concept of a “data source” that users can add to the system. An RSS feed is an example of such a data source. There is a process in the system that monitors the users’ data sources for new information. For example, when a new entry appears on an RSS feed, it is added as a memory. The AI agents then do their thing as usual.

I also replaced the MacOS speech synthesizer with the just-released Dia, which is an open source TTS model. I used the currently available 1.6B model which is quite good and a very significant improvement over MacOS’s speech synthesizer.

The dotted boxes represent the Docker containers. Very soon, I will start playing with real deployments to a public cloud infrastructure so the above decomposition will help me manage the various services separately.

RSS feed as an example of a data source

To illustrate how data sources might work, as per Ian Nutall’s post, I implemented support for RSS feeds. Connecting to a twitter bookmarks feed will be a very simple addition for a real service.

Here’s a video demonstrating the new Web app and the use of an RSS feed as a way to add links to the memory. I recorded the sound this time so you can hear the generated video summary at the end. The generated speech is better than before but not the generated images.

Here’s what happens in the video:

• The user registers an RSS feed that they control. It will contain links to articles. As new entries get added to the RSS feed, BrainExpanded converts them into memories and its AI agents start processing them. For now, BrainExpanded checks the RSS feed every 5 secs, which of course isn’t going to be necessary in production.
• I update the RSS file manually. The file is served via the Web server. BrainExpanded doesn’t know that I updated the file manually.
• In this demo, a single AI agent generates the summary of each article as a new memory. It then generates some images using the summary as prompt. It synthesizes the narration of the summary. Finally, it puts everything together in a video that the user can watch. The quality of the images isn’t great since I am using a local model. There is a lot of work still to be done to improve the quality of what the AI agents generates.
• The user can now watch the video or find the summary in their timeline.

It goes without saying that I didn’t spend time on the design of the web app. Designing web/iOS apps is not a core strength of mine 🙂

Next steps

There is still so much more to explore, especially in the set of backend technologies related to privacy, a framework for agents (or AI functions) that operate on top of the memory, integration with ChatGPT (or some other AI agent), and more. But first things first…

• Fix the iOS app now that the Web API validates the Google ID token.
• Create a Chrome plugin as another client to BrainExpanded.
• Evolve the AI agents (or “AI functions”) that operate over the memory. Improve the quality of the content they generate. Create a list of them from which users can select.
• Integrate with a note-taking/second-brain app such as Obsidian or Logseq.
• Perhaps consider a release for friends and family 🙂