(2024-10-23) Twisting The Rules Of Building Software Bending Spoonsthe Team Behind Evernote

Gergely Orosz: Twisting the rules of building software: Bending Spoons (the team behind Evernote). Today I'm happy to have the Bending Spoons team here with me. We have Luca, Francesco and Federico. When I first tweeted about Bending Spoons and how I discovered the interesting things that you're doing, I had Italian developers ping me and telling me, oh, I actually applied to Bending Spoons. They're the hottest place to be in Italy. And I asked them, when did you apply? And they told me, oh, it was like four or five years ago

The first time you came up was when you acquired Evernote, but you operate other apps and products

You correctly mentioned Evernote as one of the most well-known businesses we acquired. Meetup too. We have acquired Mosaic, which is a collection of mobile products from Interactive Corporation, and we've acquired Hopin and StreamYard, possibly the most popular recording and live-streaming suite of features of product in the world. More recently Issuu for digital publishing and WeTransfer for digital file storage and transfer. We also own and operate Remini, which is a GenAI based photo enhancer and generator

But we have been doing this for a decade and we have acquired dozens of businesses. These are just the most well-known ones.

I'd like to address an elephant in the room in terms of the perception of a Bending Spoons... they reduced the original team... So, there's this real perception of like, oh, Bending Spoons buys a company that is struggling or that wants to be sold and they just reduce the team and then they take it from there.

we do so with our own money, we're not a fund

first of all go through a phase of learning. When you are on the outside of an acquisition of a company, you don't get to see a lot of stuff you need to be inside

generally a few weeks, maybe a couple of months depending on the complexity, we find we form a pretty clear opinion and develop a pretty clear vision as to what we want to do with the product, the technology, the monetization, the organization and so on and so forth.

Often we want it to be a much smaller organization where people have a much more autonomy and ownership of projects where we do not pursue projects that are not really likely to move the needle in a major way. So, massive focus, massive leanness and autonomy.

My first reaction was like, oh, this is terrible. A company buys another company and then lets a bunch of people go, how terrible, let's say with the Evernote. But then I was kind of thinking a little bit further and thought like, hold on.
I've read all these articles about Evernote, how again, this might not represent, but what I understood, how they struggled to make a profit, how they struggled to operate, how they were burning money, how they tried this, how they tried that.
In this case, this was clearly a struggling company and to take that company to a successful company, clearly if it would've been easy, it would've clearly been done.

the world seems to be divided into two camps.
Some people think ultimately that as long as a business can survive, it should not be made more thriving, more successful if doing so means going through a painful decision such as a layoff.

And then there are people who think that as long as you act empathetically and supportively, you should try to run a business in the most effective, efficient way you can.
And that if doing so means going through hard decision like a layoff, you should do it

we try to do things the right way to build the best products we can to build the best organizations we can to treat anybody in the way out as well as we can, to monetize a product as efficiently as we can and be rational and scientific about it.

We are very transparent about it. We share a lot of stuff on our website including policies and whatnot. When we extend an offer to an applicant, we send a document called Controversial Principles where we include seven or eight values or principles we implement at the company that some people may consider controversial.

we must be doing something right, because we have a level of unwanted team member churn that's to my knowledge, unheard of in the industry. About 1% per year.

Literally a handful of people out of 450 per year, like 3, 1, 5, something like that

So, we're currently at around $700 million in revenues. We've been profitable every year since we founded the company 11 years ago. We serve well over 200 million monthly active users across a few dozen products. And in terms of team size, excluding the most recent acquisition of WeTransfer, we are a little over 400 right now.
I think 450 give or take

if you include WeTransfer, over 800 people

we built a platform that can provide services that these products can leverage as is... authentication, security, monetization, data tracking, all of these utilities

Can we talk about your AI products and specifically the AI load there? Because I know Remini, it's a very popular AI application, but how does it transform into GPUs?

average daily we do more than 2,000 inferences per second.
We can reach peaks of 8,000 per second, and that turns into, if you look at the GPU demand that we need to satisfy in order to fulfill these inferences, we're talking about around 4,000 GPUs

having grown through our own cash flows and a little bit of debt from traditional banks has forced us to be extremely efficient and thoughtful about the allocation of resources, more autonomous and independent.

loans from traditional banks

Luca Ferrari: Right. When I graduated in Copenhagen, Denmark. Studied engineering and with two friends at the time, we started a company called Evertale, and the vision was to create a self-writing diary of a user's life using AI.
So, we would install this app and it would collect data from the phone transparently then through AI, and at the time we weren't even talking machine learning

we were left with about $40,000 of VC money. In that case, we had raised $1 million

the VC firm told us they sold their shares to us for $1 because they didn't want to go through the pretty cumbersome liquidation process.

it actually worked reasonably well, but it was a commercial failure

started in Copenhagen in 2013 with that $40,000, the same three people who started Evertale, plus two people who had been hired at Evertale and turned out to be particularly capable.
And so we asked them to co-fund Bending Spoons

we invested in the first acquisition $10,000.

that one was an app, an iOS app to personalize your keyboard

we tried to improve it ultimately turning that $10,000 investment into say 20,000, and then 20,000 gets you a slightly bigger, more promising acquisition, and if you're good, you turn it into 40, then the compounding is a pretty powerful force over time, exponentially.

we acquired Splice, a mobile video editor from GoPro. That was the first time we acquired an asset from a large structured seller as opposed to single individual developers or small teams around the world.

we came to know that Evernote could potentially be acquired around late 2022. We jumped at the opportunity

the acquisition actually closed very early January '23, and we jumped into the organization

at the end of this process we try to come up with a vision and the vision spans the gamut. Organization, technology, user experience, monetization, marketing, everything.

we acquire these companies to own and operate them forever. So, we think with the longest timeframe we can. And so that vision embeds that sort of long-term thinking in it. And then the second phase starts, which is where we try to close the gap between the status quo and that vision

it was migrated at a certain point from the more traditional data, let's say bare metal servers to the cloud space.
And that migration was of course supported by the cloud provider. It was made in a one-on-one fashion, meaning that it wasn't really happening ... Evernote didn't, let's say reshaped the infrastructure or the architecture, making it cloud-native.

we found ourselves with a monolith, a huge one with a Java 11 monolith, and it was running on virtual machines on Google Cloud, but they were sort of manually provisioned.

Users' data were sharded among all of these machines. That sharding was not even after thorough analysis, doesn't even have aspiring to be even meaning that certain machines have heavier loader than others, significantly heavier, and sometimes that caused it also, a lot of interventions, a lot of maintenance

We had to realize that on-calls were actually very needed.
We run several products. What we do generally, we try to make them reliable enough to not need an on-call program.

I think a broader higher-level principle we follow religiously at the company and it's probably one of the main drivers for these views on the on-call programs is call it radical simplicity. (Simplest Thing)

it's also policies could be made simpler, and we have plenty of examples of systems that I think were reasonably simple to start with and made radically simple because of this approach. One being pay.

for many, many years we've had fixed pay for everybody, a hundred percent fixed. There's only a salary and people can choose to receive a percentage of their salary in equity if they so wish.

What is Evernote on these days in terms of the technology stack

all the data user, the users had their data stored on the virtual machines, actually the databases, they were all deployed locally on the virtual machines. That was one of the biggest weak point in our opinions.
The first thing we did was migrating all the data from the shards from the virtual machines to a managed database structure.

after some analysis, we found out that the best way moving forward was to move from the monolith to a microservices architecture.

One of the core projects also that we needed to take care of to do such a move as changing the interaction mechanism between the clients and the backend substantially before the clients were polling continuously bombarding-

the clients also had to, let's say, embed a heavy, let's say quite a big layer of logic just to take care of all the pollings and all the combining the mutations that were coming back from the backend

what we had to do was to move from this polling mechanism to an event-driven communication.

we moved all the logic of combining all the mutations and making sure that the end state is unique and clear to the backend.

How did you get to this really fast shipping cadence?

we do have some bigger tracks that take longer to develop

But on the other hand, as we understood better what our customers wanted, we tried to prioritize those things that people wanted, but were also easy to build in a sense.

A few examples are, for instance, collapsible sections inside the note editor, Evernote was missing this feature and we were able to build it in a couple weeks.
Other examples are slash commands through which you can hit slash and select an item to add to a note.

the easy and visual things and then I'll do the hard and long thing

We're doing both. Showing both externally and internally what we're building is easy when you are shipping a product feature.

what helped a lot shipping fast was also to completely review the CI/CD pipeline. I think it's worth saying that we made huge advancement in that regard.

Some of the pipelines, most of them were on self-hosted Jenkins, just to give you a sense.
And we moved to CircleCI with our orbs, the orbs that we built in the years that we polished a lot, we know what to expect and what not.

we're closer than ever now to the complete dismissal of the monolith

it's a villain, something to be afraid of really to fear.

And as soon as we get to the finish line, I believe that's going to be one of the most exciting moments for the Evernote engineering team because at that point we'll be ready to finally reshape the system entirely to make it much more efficient.

there is, in our opinion, many more microservices that actually needed ... what's the trade-off with microservices, you need them of course, but sometimes can get out of hands, it can be hundreds.

We want to consolidate the responsibilities and the numbers of microservices out there in the architecture now, it's just not the right moment because we first need to get rid of the huge legacy.

we recently released what I believe from my tests at least, is the best AI transcription feature available on any note-taking product

I was honestly surprised myself by how accurate it is

we have different business units, Evernote being one, and these business units have dedicated management teams and resources such as software engineers, data analysts, scientists, product designers, so on and so forth, depending on the needs specific of that product and business.

And then we have a shared platform, including several platform teams. And what the platform does is it supports and serves the different business units with tools, services that are relevant, generally speaking to most of them.

it's important for us to not be prescriptive in the tools and the methods that we want to follow across the different business units or across the different teams. Tools are tools that are meant to achieve a certain help, to achieve a certain objective. Processes have the same objective.
Let's say some teams are more used to follow certain processes than other. We give the team the freedom to choose whatever they believe is the most effective approach.

software engineers really care about the product and the end result of it. So, doing white-boxing QA is something that they take very seriously.

But on top of that, then of course black-box QA instead is done by the QA or testers. We make sure that everything that can be tested by software engineers before is properly tested.

If the core part is about the UI or if it's about some critical features, I'd say we develop a lot of end-to-end tests for sure.
We try to reach the right level of coverage.
We try also to be pragmatic

I believe also in terms of observability and monitoring, in addition to QA, something that we have spent a lot of effort is to be sure to tailor metrics or what we test and how in such a way that we're sure that we have the highest coverage, but we're also very careful to not trigger false positives.

We have a team, we call it Foundations Technology. It builds all the tools, expertise and knowledge to make sure that we have the right developer experience, that we have analytics tools, monetization, user acquisition tools, and we make sure that these utilities can be used by the business units if they want.

That has proved to be extremely, extremely useful because when you have, let's say, when your data warehouse is built in such a way that all the products produce data with the same kind of semantic and format, every tool that you build on top of it is out of the box ready for all the other digital products when you build it for one.

if a business unit has a need that's incremental to whatever shared technologies available, they can improve it. And so they can pass up, let's say toward the platform improvements that they can then be propagated to benefit any other business unit in the future.

the most popular back-end language that we use is Python. We use FastAPI, for instance, to ship APIs. We also use TypeScript a lot, especially because we've been investing a lot in making engineers, let's say full stack as much as possible.

We use Swift and Kotlin to build native iOS and Android apps typically, but we also rely on React when it makes sense. We use Electron.

We always have favored hiring new graduates or inexperienced people. And there are two reasons for it. One is when you hire, there's a trade-off between a few things, certainly talent and experience. And so the more you want to optimize for talent, the less experienced people will be on average and vice versa.

Talent we cannot teach.

That means we need to invest heavily in training coaching.

The second reason is our approach, our culture are so unusual

it's proven challenging for highly experienced individuals to adjust to them sufficiently quickly

We don't have any titles. So, if you're hired as a software engineer, you are and always will be, as long as that's what you do called software engineer internally. And you can choose your own title for whatever external reasons you like, your CV, LinkedIn. We only ask that you don't do something that would embarrass us, like calling yourself the CTO if you're a new hire, something like that.

All managers are simply called leads

We have a few levels. There are very few. The difference is pretty substantial. And so we don't try to resolve finer deltas in your contribution

we look for people who are very strong problem solvers, who can learn very quickly, people who have a strong sense of ownership. So, they really want to own their objectives, their work. You don't have to check on them.

And third, I would say a trait we are really big on is team spirit.

if those things are true of someone and they really want to ... they see their role in an expansive, horizontal way that they're not in this for just becoming the most expert Python programmer in the world, but they want to be a great 360 degree engineer who understands the business impact of what they do, who has an interest in that. I think that such a person will probably do really well at Bending Spoons and love it.

You could say probably that seven years ago we were the company everybody's trying to build now.