Learnings over the 6-year Journey at Arc/Codementor
Six and half years ago, I joined Codementor as a full-stack developer. I was the seventh team member of the company, located in a co-working space, Garage Plus. Years after, I’m about to graduate from the team as the Director of Engineering, with tons of great memories, experiences, and friendships. In this post, I’m going to share my gratitude and learnings over the beautiful journey.
The Overall Diff
Back in 2015, we had one product, Codementor. It was a Rails server-rendered website using Angular1 as the JavaScript framework. Now we have two products working in parallel, based on multiple services communicating with each other. Our user base grows by 30x, with the team size growing by 7x.
Together, we have built a set of well-functioned development processes fitting our current team size and communication structure from scratch. We have automated testing of different levels, CI/CD, regular process review discovering the missing parts, dedicated resources to handle Non-Functional Requirements, among many others that make our lives easy and productive. And most importantly, we have a solid team taking care of these and discovering unknown unknowns.
How to 10x a team
When I first joined Codementor, I wished I could 10x the team’s productivity without any title. I’m not sure if I’ve achieved this goal as measuring productivity is itself a challenging task. But I think there is still something I can share, given we survive the company’s growth as a team.
High-Leverage Items
One critical factor is identifying and working on the “high leveraged” things to produce the highest impact with minimum efforts. For example, introducing automated testing to the team is high leverage because it shortens each team member’s feedback loop. The more team members in the group, the higher the influence is. Working on a challenging problem might not be one. But mentoring other team members so that they can also solve it is a high leveraged one. For more details about high-leverage items, I found the book “The Effective Engineer” quite insightful.
Culture
Among all the high-leverage items, shaping the culture might be the most critical one. The definition of the culture here is “The way a group operates without any rules or guidelines.” We can’t have rules for everything. Those not covered by any rules, such as how we treat each other, how we communicate, and how we give/receive feedback, are governed by the culture.
There’s no one-size-fits-all culture. You want to identify the ones most suitable for the team and the ones you like. This part is sometimes subjective — leaders and senior members might have larger weights on this, but I think it’s inevitable. Take Codementor/Arc Dev Team, for example. Part of cultures is to make decisions rationally, always identify the pros, cons, and probability, and be assertive and open-minded at the same time. It somehow reflects the personalities of me and some early team members.
To shape a culture is far from an easy task. Leading by example is one of the effective ways. As a leader, if you want the team to share direct feedback, encouraging the team to challenge you in public and showing sincere appreciation after that is far more convincing than just repeatedly asking the team to do so but emotionally reacting when someone does.
Hiring
Another fundamental high-leverage thing is good hiring, especially for early team members, as they have much larger weights over the culture.
Most hard skills, be it programming languages (even spoken languages :p), library, frameworks, can be learned relatively quickly. But for personalities, it usually takes years to change, if possible at all. However, knowing candidates’ personalities well during hiring is challenging. Sometimes identifying the poor hirings and letting them go smoothly is as important as hiring.
How to work with the Growth Pain
User base and traffic
There isn’t one specific trick to deal with the user base and traffic growth for our case. It’s always case by case. There are so many tricks on Google for scaling your service, but I think the hard part is how to diagnose our specific situation and make sensible trade-offs. The way we adopt is analytical decision-making:
- List out all the pros and cons with each associated probability.
- Model the qualitative factors into quantitative ones if possible.
- Calculate the expecting value base on the above to make a decision.
As an example of the above, we once discussed whether we should stay with Heroku or move to Kubernetes on AWS (EKS.) With our experiences on Kubernetes considered (which was not that much), we modeled into quantitative values (money) the learning time combined with the average developer rate, the revenue loss introduced by the potential outage combined with its probability. And we compared it with the cost difference between Heroku and AWS. With that, we decided to stay on Heroku and have a Heroku cost threshold to move to AWS.
One thing that is specific to us might be the Legacy Code Issue. Cleaning up/refactoring the legacy code/architecture helped us serving more traffic while staying on our existing architecture. One example is that we migrated our web app from Rails server rendering to React Single Page App (SPA.) Along the way, we refactored how we structure our back-end app using Service Object Pattern, thus making the queries easier to maintain and more efficient. This move directs some loads from our server to users’ browsers while introducing more services and other architectural tweaks, which was a consciously-made trade-off.
Dev team size
The first lesson of working with team size growth is to be cognizant that the communication effort is O(n²). Anything from the pull request reviewing time to the NPS score of the product can be affected by the team size. To make things trickier, most of the challenges lie in our unknown-unknown quadrant — we don’t know what we don’t know. Fortunately, we found some approaches helpful:
- Processes that surface potential issues
We cannot eliminate all the issues, but we can surface them faster, thus reducing their impact. In essence, it’s to shorten the feedback loop, just as the purpose of CI/CD and unit tests.
For our case, we have periodical reviewing meetings to discover the potential problems. Conceptually, it is similar to the Scrum Retro but with a broader scope. We referenced Spotify’s health check model and tweaked it according to our situation. Sometimes the issues raised are not necessarily the root causes but just syndromes. It requires some discussion or creativity to find the root cause.
2. Be aware of Conway’s law and team structure.
In programming, sometimes, the complexity is due to the wrong abstraction. If you structure your code the wrong way, a simple change might require lots of message passing hence couple the code. Similarly, when the teams are structured poorly, each team will no longer be autonomous, and a simple change requires communication among many groups.
3. Communication with the different functional teams
We had no designers, no product managers, no marketers, and only developers at the company’s beginning. Each developer made the decisions end-to-end. Over time, we now have a full-fledged Design, Product, and Marketing team. We can work deeper on each part, but each team’s incentive also needs to be aligned. The way we think and operate changes a lot over the course. Learning to work and communicate with groups with different thinking frameworks is valuable for all of us as a company.
From Senior Developer to Director of Engineering
As a senior developer, I was privileged enough to support the team. And I’m lucky enough to meet these talented team members that teach me how to be a good leader. I think it’s essential to position myself as the one to “support” rather than to “lead.” For me, the primary learning is to shift my goal from “being the most capable one” to “making the most capable ones.” This shifting implies many things, including mentoring, adequately delegating, providing more context, incentivizing the team by paving the career paths for them, and many more.
Thanks
Special thanks to our CEO Weiting, COO Gage, and Head of Growth Tongtong. Thank you for making this beautiful journey possible. Thanks for letting me join the team, trusting me, and giving me ample space to do whatever I believe is right. Thanks for letting me experience the growth of a startup and become a better person together with you. I’ll never forget the time when we camped in the house in Sunnyvale building CodementorX.
And each of the Dev team members, thank you for working with me to build such a strong team that we are all proud of and enjoy working within. Thank you for every single feedback you give me, be it positive or negative. Thank you for teaching me how to lead a team and be part of it at the same time.
Also, to our Product Leads, Joyce and Yutang, thank you for letting me build the beautiful products with you. With the trust among us, I can always share every idea in my mind without afraid of being too dumb. Thank you for selflessly sharing your perspectives and pointing out my blind spots.
Last but not least, thanks to our Head of Design Young, VP of Sales Fossi, Head of Marketing Christine, and HR Georgina. It’s my honor to work closely with you, and I learned a lot from you. Thanks for growing the company in each dimension; it can never be where it is without anyone of you.
And for all the above, thanks for giving me the friendships and the sense of belonging.
Oh btw…
The team is actively hiring for almost every position. See here for the openings!
