Engineering blueprint: how we cut matching latency by 95% using vector search

To provide a best-in-class experience, a talent platform must do more than just list profiles: it must understand the nuance of a client's technical needs and deliver the right expert instantly. Back in 2023, as Malt scaled to over 700,000 freelancers, our original matching engine hit a "latency wall," where complex searches could take up to a minute to process. To solve this, our engineering team rebuilt our recommendation system using a "Retriever-Ranker" architecture. By leveraging high-performance vector databases, we successfully reduced our response times from 60 seconds to under 3 seconds without losing the precision our clients expect.

In 2021, our system relied on a "monolithic" model, meaning one large program did all the work at once. While accurate, this model had to evaluate every single project–freelancer pair. As our community grew, this became mathematically impossible to do quickly, leading to slow response times that hindered the user experience. We needed a way to search through hundreds of thousands of profiles in the time it takes to blink.

To balance speed with accuracy, we split the matching process into two specialized steps:

The retriever’s job is to quickly narrow down the pool of 700,000+ freelancers to the 1,000 most likely candidates.

• The tech: We use a bi-encoder architecture.
• How it works: Think of this like a library where every book is already indexed by its meaning. We turn freelancer profiles into "vectors" (mathematical coordinates in a digital space). When you post a project, we instantly find the profiles that are mathematically close to your requirements.

• The benefit: This step is incredibly fast because it uses pre-calculated data.

Once we have the top 1,000 candidates, the Ranker takes over to ensure the final order is perfect.

• The tech: We use a "cross-encoder" model.
• How it works: This model is more powerful but slower. Going beyond simple keyword matching, it performs a deeper semantic analysis. It looks at the project and the candidate together to understand deep context, such as specific framework versions or industry experience.
• The benefit: Because it only has to look at 1,000 people instead of 700,000, it provides gold-standard accuracy in just a second or two.

The retriever needs a specialized home to store these vectors. We tested several industry-standard databases to find the best fit for our specific needs, such as geo-spatial filtering (finding experts in a specific city).

In a benchmark conducted in 2023, while Elasticsearch is a tool we know well, its speed was too low for real-time recommendations. Qdrant was the clear winner, offering a 30x speed increase over Elasticsearch while maintaining high accuracy and the ability to handle complex filters like freelancer availability and location.

By moving to this new architecture, we achieved more than just a technical upgrade:

• 95% latency reduction: Our p95 latency (the time it takes for the slowest 5% of searches) dropped from 60 seconds to under 3 seconds!
• Better matches: A/B testing confirmed that we could achieve these lightning-fast speeds while maintaining match quality equal to our previous system, and crucially, this architecture paved the way for advanced ranking models that are now driving higher conversion rates. 
• Custom AI: We confirmed that training our own AI models on Malt’s specific data, rather than using off-the-shelf generic models, was essential for understanding the unique language of the freelance market.

This architecture ensures that as Malt continues to grow, our ability to connect you with the right expert remains near-instant.

Ready to scale your own AI initiatives? Whether you need a specific expert to clear a backlog or a full team to build your RAG architecture from scratch, Malt connects you with the top freelance engineers in Europe. 

Explore more engineering insights. This architecture is just one part of how we are building the most efficient freelance marketplace in Europe. Read more about our approach to Data Science, Engineering, and Product on our full engineering blog.