The open data portal of the Spanish city of Valencia provides real-time traffic data for both cars (and other motor vehicles) and bicycles. A series of electromagnetic loops distributed throughout the city are capable of measuring the intensity of vehicles passing through them. This information is published on an hourly basis for motor vehicles and approximately every 30 minutes for bicycles. With that information, valencia-now is able to build these two maps representing current traffic intensity:

The bicycle traffic is also represented in the same way, as well as the air quality data obtained from various measurement stations located in different parts of the city. In this case, we are representing the air quality index, a value that can range from 1 (hazardous) in color purple to 6 (good) in color blue.

If I were to stop at that point, I wouldn't be adding much additional value to this data, since the open data portal itself already provides some simple visualizations of the current state of traffic and air quality.

However, valencia-now is also collecting and storing this information which allows me not only to see the current status but also the evolution.

With that historical data, I am also able to provide aggregations such as the following one that shows how the air quality in the city centre improved just a day after the Fallas finished.

In the following days, I would like to create some additional blogposts. I want to explain technical details about the data ingestion and the data transformation processes (spoiler, tinybird made this extremly simple) and also about public open data iniciatives.

At some point in time I would like to make the collected data available for everyone, but this will require some work.

🗨️ As always, feel free to contact me (unmonoqueteclea@gmail.com) for whatever doubt, suggestion or idea you may have. Happy to receive your feedback!

Almost all public podcasts have an associated RSS feed that contains metadata about every episode, including a link to the audio file. As an example, this is the feed from the Ted Talks Daily podcast.

Voilib collects and stores metadata from the list of podcasts feeds manually configured by the application admin. For each episode, it stores in a SQLite database things such as the title, the description, the language or the duration.

When I want to check if new episodes were published, I just need to run this in a command line (although I have a cron job configured to run it twice a day):

voilib-episodes --update

If you want to dig into the code, check feed.py and crawler.py modules.

The podcast episodes are transcribed using Whisper, an Open Source Speech Recognition Model developed by OpenAI. Voilib effectively leverages Whisper, utilizing the whisper-jax library. This particular implementation of Whisper boasts exceptional efficiency, showcasing speeds up to 70 times faster than comparable alternatives.

However, most whisper-jax optimizations are predominantly GPU-focused (that Voilib is not using yet), I must admit that I have not observed significant improvements in CPU performance when compared to the official implementation. Nevertheless, the accuracy of this model is nothing short of remarkable, even for the small version with "just" 39M parameters. It is truly astonishing to think that a few years ago, having an Open Source Speech Recognition model as swift and accurate as this one would have been deemed nearly implausible.

When I want to transcribe the new episodes from the last 3 days, I just need to run this (I also have a cron job configured to run it):

voilib-episodes --transcribe-days 3

For curious minds, you can check transcription.py module.

The process of generating episode transcripts involves breaking them down into fragments, each comprising approximately 40 words. Voilib employs the sentence-transformers Python library to calculate the embedding of each 40-words fragment, effectively transforming it into a 384-dimensional vector of floating point numbers.

To shed light on the concept of embeddings, they can be defined as lower-dimensional spaces that allow for the translation of high-dimensional vectors. The goal of embeddings is to capture semantic similarity, ensuring that inputs with similar meanings are placed closer together in the embedding space.

The sentence-transformers library stands out as the optimal choice for creating text embeddings. This is the description from their website:

You can use this framework to compute sentence / text embeddings for more than 100 languages. These embeddings can then be compared e.g. with cosine-similarity to find sentences with a similar meaning. This can be useful for semantic textual similar, semantic search, or paraphrase mining.

The framework is based on PyTorch and Transformers and offers a large collection of pre-trained models tuned for various tasks. Further, it is easy to fine-tune your own models.

The model Voilib is using to calculate the embeddings is multi-qa-MiniLM-L6-cos-v1. This model have been specifically trained for Semantic Search with 215M question-answer pairs from various sources and domains.

You can find the code for Voilib's embedding calculation in the embedding.py module.

Those calculated vectors are then stored in a vector database, one of the main components of the system. More on this in the next section.

Each episode's embeddings are meticulously preserved within a vector database. In recent months, we have witnessed a surge in new technologies specifically designed for storing embeddings. These innovative solutions empower us to efficiently store embeddings and seamlessly query them using cutting-edge Approximate Nearest Neighbor search algorithms.

During the initial stages, Voilib relied on Meta's FAISS library for embedding storage. However, to enhance the system and include additional metadata for each embedding (that can be also used to filter queries), I decided to migrate to qdrant, a higher-level solution. With qdrant, we achieve the ability to incorporate supplementary metadata while seamlessly managing embeddings. As part of this evolution, Voilib now operates its own instance of the qdrant server, further ensuring data autonomy and control.

For every user prompt, Voilib performs a swift calculation of the corresponding embedding and efficiently queries the vector database. This process enables the system to swiftly identify and return the most relevant results, ensuring a seamless and satisfying user experience.

Of course, there is another command to calculate and store embeddings from all pending episodes:

voilib-episodes --store

More, in the vector.py module.

Please feel free to reach out to me at unmonoqueteclea@gmail.com with your thoughts, suggestions, or any inquiries.

I am eager to know which podcasts you would like to see available on Voilib. Additionally, if there are any specific features that you believe would enhance your user experience and make your life easier, please do not hesitate to share them with me. It's a fantastic opportunity to contribute to the open-source community.

If you have been considering hosting your own instance, I would be thrilled to support and guide you through the process.

I look forward to hearing from you!

Introducing now Joe Shmoe, the HR leader responsible for reviewing Jane's captivating cover letter. In the fast-paced world of talent acquisition, Joe faces the daunting task of sifting through an impressive volume of cover letters daily. His goal is to identify the standout applicants worthy of an interview. To accomplish this challenging feat efficiently, Joe relies on the cutting-edge capabilities of ChatGPT to extract the key insights and main ideas from each cover letter:

Could you please extract the 3 o 4 main pieces of information from this cover letter as bullet points?

And this is the answer he obtains from ChatGPT (find it here):

• Highly skilled and motivated Software Engineer with over 10 years of industry experience
• Led a successful migration project from AWS to on-premises infrastructure, resulting in an annual cost reduction of $400,000
• Extensive experience in working with sensitive healthcare data and leading a team of 50 Software Engineers in the development of a Hospital Management System for 10+ hospitals in the US
• Successfully developed and launched a meditation app for iOS and Android platforms with over 5000 monthly active users.

Does it sounds familiar to you too?

LLMs are beginning to change the process of content generation but also content consumption, giving rise to some rather paradoxical scenarios like the one I described.

• When it comes to content generation, humans contribute valuable information (signal), while LLMs (usually) decorate it with some irrelevant elements, effectively adding noise.
• During content consumption, humans trust LLMs to effectively distinguish and eliminate noise from the essential information.

A fascinating feedback loop appears in this new paradigm, wherein efforts are made to enhance the intelligence of LLMs by using text sourced from the Internet, which in some cases might have been generated by other LLMs (making the signal to noise ratio lower on each iteration).

I'm intrigued by the future of Language Models and the potential revolutionary applications they might enable but I wonder whether we'll see a resurgence of encyclopedias as the primary source of reliable information with a high signal to noise ratio.

To keep a repository with all the dotfiles I am using GNU Stow. On its website, GNU Stow is defined as a symlink farm manager. It helps me to manage automatically symlinks between $HOME (or any other place that can contain dotfiles) and my dotfiles repository. I included a makefile in the repository to automatically create all the needed links using this tool.

The biggest part of my dotfiles repository is my Emacs configuration in emacs/.emacs.d. I have a modularized Emacs configuration, keeping .el files for things like agenda, theming, keybindings, etc. I try to add explanations for everything I add there, so I hope they can be useful to you.

There are always private things in your dotfiles that you don't want to show.

For instance, in ssh configuration, there are some entries that I want to keep secret. The solution was easy, I just need to add a new config.secret file with those entries. That file will be included in my .gitignore file and, to ensure I can access those entries, I added the following to my main config file:

Include config.secret

All your secret entries will be automatically imported!

I also keep some gitignored *.secrets.sh* files that include APIs keys and similar things.