zerodha case study pdf

• Thought Leadership
• Bharatiya Vidya Bhavan's SPJIMR

The curious case study of Zerodha's blue ocean strategy

Founded by Nithin Kamath, the online discount brokerage company has changed the dynamics of retail stock investment in India. Here's a look at how the blue ocean strategy worked and what it needs to do to face new challengers

Related stories

Competing with each other about who has more in the bank is becoming less cool today: Nikhil Kamath

The good, bad and ugly of stock trade training courses

Bull or bear, long-term stock market investors win

How zerodha attracted non-customers.

[This article has been reproduced with permission from SP Jain Institute of Management & Research, Mumbai . Views expressed by authors are personal.]

• " class="general-icons icon-sq-googleplus popup">
• " class="general-icons icon-sq-youtube">

UK Elections 2024: British-Indian women who won Parliament seats

Photo of the day: UEFA Euro 2024 champions: France wins

From NEET-UG paper leak to paradoxical Indo-China relationship: Our top stories of the week

First stretch of smart highway opens in the US

Swiss collection of Impressionist masterpieces shines under a new light

Air pollution drives 7 percent of deaths in big Indian cities: study

New cars have gained nearly 400 kg in seven years

From Aerostrovilos to Bellatrix Aerospace: Deeptech startups, one year on

T20 World Cup Victory Parade: A celebration like no other

Linking CEO Compensation with ESG markets is the worst idea: Rajeev Peshawaria

Photo of the day: Britain chooses Keir Starmer

Need a people-first approach for a sustainable era: Jagjeet Singh Sareen

How India Eats: Heat, unseasonal rains, virus infestation push veg thali prices higher in June

Strategic alignment in the age of AI: The 7 foundations of competitive success

Revenge savings: New trend among Chinese youth

Zerodha’s Emergence: A Bootstrapped Unicorn’s Meteoric Rise

The first name that comes to our mind when we discuss discount brokers in India, is Zerodha . The name has been coined from a combination of ‘Zero’ and ‘Rodha’(a Sanskrit word for barrier). With a humble beginning on August 15, 2010, Zerodha was co-founded by Nithin Kamath (CEO) and Nikhil Kamath (CFO).

Z erodha, India’s leading online brokerage platform , has been shaking up the traditional stock brokerage industry since its founding. With its innovative business model and use of technology , Zerodha has been able to drastically lower costs for customers, making stock trading accessible to a wider range of people. But the company’s impact goes beyond just making trading more affordable, it has also been a catalyst for change in the industry, inspiring others to follow in its footsteps and adopt similar strategies.

One of the peculiar things that people are intrigued to know about Zerodha is that even though it has been bootstrapped and didn’t raise any funding from any investors , it managed to enter the “ Unicorn ” club. Zerodha entered the Unicorn club based on its self-assessed valuation of $1 billion. Nithin Kamath, in one of his interviews with Raj Shamani, mentioned that the period of 2010 was not favorable for starting a stock broking firm. A few of the reasons being, the brokers lost a lot of money and were affected by the financial crisis. The trading activities were reduced drastically. Also, he felt less awareness of VC in India. So if they would have met professional VCs in 2010 and if the VCs would have given money, the company may have probably raised funding for the firm .

Let’s try to understand how Zerodha disrupted the stock brokerage industry, what is its business model and how it generates revenue!

Before Zerodha, Nithin Kamath and Nikhil Kamath co-founded a firm, called Kamath Associates back in 2006–2010. During this, they realized the high costs and lack of transparency in the traditional stock brokerage industry. Zerodha’s disruptive idea was to eliminate the traditional percentage-based commission and introduced a flat fee structure for trading , making it more affordable for the customers.

Zerodha seemingly follows — “low margins, high volumes” as their business model. The company saw an opportunity to use technology to disrupt the status quo and make stock trading more accessible to everyday investors.

Reasons for the high profitability of the firm

• Though they have a low fee of Rs. 20, the number of transactions per day is quite high in volume. This in turn gets more revenue in total for the company
• They have managed to keep their operational costs low because of their online platform/technology, aiding them with high-profit margins.
• Just the way, Zerodha is a bootstrapped company, they also believe in a “No-marketing” policy, which in turn saves them a ton of money. This is one of the reasons that they believe in mouth-to-mouth marketing which is possible only if their products are well-developed and provide a better customer experience.
• Zerodha offers various products and services related to financial investments and technology.
• Rainmatter , an initiative by Zerodha, funds and incubated innovative Indian Fintech startups.

Zerodha offers the following products and services:

Financial investments.

• Stocks : Trade stocks for delivery or intraday on over 5000 stocks listed on National Stock Exchange (NSE) and Bombay Stock exchange (BSE).
• Direct Mutual Funds : Invest in over 2000 direct mutual funds directly without a distributor.
• Future & options : Trade metals, oil, and Agri commodities on MCX and stock and index futures and options on NSE.
• IPOs : Invest in companies listing on the Indian exchanges with an IPO (Initial Public Offering)
• Gift Stocks : Introduce your friends and family to the habit of investing for the long term by gifting them stocks, ETFs, mutual funds, and gold bonds.
• Fixed Income: Invest in Government securities bonds, sovereign gold bonds, etc.
• Currency Trading: Trade in currency derivatives.

Technological

• Kite : is a web and mobile platform that offers access to trading and investment products, including equities, derivatives, currencies, and commodities, with streaming market data.
• Kite Connect API : An API for developers
• Coin : A platform for direct mutual fund investments, free.
• Varsity : A free online education portal that provides educational content on various investment and trading-related topics.
• TradingQ&A : The most active trading and investment community in India for all your market-related queries.
• Console : A platform for account management
• KillSwitch : A Service for risk management for retail traders
• Sentinel : A cloud-based price alert engine

Funding and incubation by RainMatter

Investments, main revenue generation sources.

• Rs. 0 or free equity delivery — All equity delivery investments (NSE, BSE), are absolutely free — ₹ 0 brokerage.
• Flat ₹ 20 or 0.03% (whichever is lower) per executed order on intraday trades across equity, currency, and commodity trades. Flat ₹20 on all options trades.
• All direct mutual fund investments are absolutely free — ₹ 0 commissions & DP charges. Detailed information about the fees: https://zerodha.com/charges/#tab-equities

One of the important points to note here is, though they have kept the fees quite low, they ensure that the quality of their products is not compromised.

Let’s look at some numbers over the years

It’s not just the product or the business strategy that has helped them to build their reputation. Their ideology of customer-centric policies helped them to build better relationships with their customers.

The company is working more toward its nudge feature that will support traders in taking informed decisions about their investments. It warns investors before buying penny stocks. It launched a killswitch , that will instantly disable trading for investors that constantly make losses. This disabling will lead to fewer trades making losses, but it blocks more profit for the company. But they consider minimizing customer losses over making more profits for the company.

Challenges faced by the company

• The company is fully dependent on the underlying stock markets. They anticipated a market slowdown which would lead to the slowing down of the company’s growth.
• Their OMS(Order Management System) faced certain technical glitches because of heavy trading activities. This leads to a reduction in customer satisfaction at times. But they are ensuring that their technology is up-to-date and working on building robust systems.
• Though the platform provides various products and services, they still do not have features like daily reports, news alerts, etc.

Zerodha’s various strategies as mentioned till now, viz. educating their customer first for free, utilization of behavioral economic concept nudge theory of killswitch, no-advertising, low-cost, etc has successfully helped the company to showcase a strong competitive positioning.

As of 31st March 2022, Zerodha constitutes around 17.42% of the total market share of the active clients registered on the National Stock Exchange. It has over 62.77 lakh active customers compared to a total of over 3.50 Crore active clients of all stockbrokers on the NSE.

Zerodha has been able to overcome various challenges and continues to grow. The company has been able to achieve this by staying true to its mission of making markets accessible to all, and by continuously innovating and adapting to the changing market conditions.

Zerodha has also been awarded as the

• Economic Times Startup of the Year — 2020
• NSE, BSE, MCX — Best Retail Brokerage Award — 2018, 2019, 2020

Zerodha’s business model and use of technology have disrupted the traditional stock brokerage industry in India, making stock trading more affordable and accessible to a wider range of people. The company’s success has inspired others to adopt similar strategies and has had a significant impact on the industry as a whole. Despite facing challenges, Zerodha has been able to overcome them and continue to grow, and it will be interesting to see how the company will evolve in the future.

Additional References

• https://zerodha.com/
• https://startuptalky.com/zerodha-trading-services/
• https://tradebrains.in/brokers/how-does-zerodha-make-money-a-look-into-its-revenue-model/
• https://zerodha.com/z-connect/featured/12-years-of-zerodha
• https://www.adigitalblogger.com/research/zerodha-valuation/
• https://www.youtube.com/watch?v=E4RO8Mstaac
• https://www.truebeacon.com/
• https://rainmatter.com/

Note: This article is not a sponsored article. It is purely an interesting case study.

Feel free to share your thoughts in the comment section. Follow and subscribe to me if you wish to receive notifications for similar articles.

If you are unable to comment for any reason, feel free to drop me a message on Linkedin .

Written by Kush Shah

Data Platform Engineer | Designing and building Internal Developer Platforms | Tech | Startups | Business | Music | Fintech | Product | Human Psychology | Books

Text to speech

• Español – América Latina
• Português – Brasil

Zerodha: Improving the retail investment experience with seamless collaboration on Google Workspace

About Zerodha

Founded in 2010, Zerodha is one of India's largest stock brokers, serving millions of users. The company offers investors attractive brokerage rates for all types of investments, including stocks, mutual funds, and commodities. The name Zerodha borrows from the English word, Zero, and Rodha, which means barrier in Sanskrit. Zerodha removes barriers for traders by charging a flat fee of 20 Indian Rupee, the equivalent of 27 cents, rather than a percentage commission in investment size. The company donates a portion of its profits to the Rainmatter Foundation, an NGO dedicated to combating climate change.

Tell us your challenge. We're here to help.

Zerodha enables employees to work remotely during the covid-19 lockdown with google workspace, enabling secure access to enterprise apps, while enabling real-time meetings anywhere with google meet..

• Collaborating on Docs for quicker development of blogs and company newsletters
• Transitioned 1,000+ staff with minimal effort to full remote work overnight using Google SSO with 2FA
• Onboards millions of users in 4 weeks Automate document verification with Vision AI to speed up application process

Enables new account opening in minutes instead of days

The rise of smartphone ownership in India in recent years alongside the pandemic has brought about a surge in retail investing. More people are choosing to enter the stock market through their mobile devices due to the ease of access. In fact, the Indian National Stock Exchange (NSE) reported that mobile trading jumped from less than 7% in 2019 to more than 19% in 2021. Zerodha , one of the largest online brokerages in India, experienced a double-digit growth in customers every month since the outbreak in 2019.

Featuring low brokerage fees and easy-to-use apps, Zerodha simplifies trading for anyone with a smartphone. By reducing manual tasks, the company keeps margins low and invests in developing digital solutions such as its flagship trading platform, Kite, and its mutual fund investing platform, Coin.

When Zerodha began operations in 2010, the team was eager to eliminate cumbersome tasks related to legacy solutions, such as managing mail servers. According to Vishnu Sudhakaran, full-stack developer at Zerodha, Google Workspace was the obvious choice. This was because Google Workspace was easy to set up for employees as they were already familiar with the Google ecosystem and did not require any training.

“Zerodha's guiding principle is scalability. Our trading platform grew to millions of users with the help of a small tech team,” says Vinay Kumar, AVP Technology at Zerodha. “Rather than managing email accounts and servers, we want engineers to focus on building digital solutions that our customers need.”

Team collaboration in the new normal with Google Workspace

Google Meet is now an integral part of life at Zerodha to help colleagues stay connected. Virtual meetings are a fairly new practice at Zerodha, as it only started to become a norm since the COVID-19 pandemic. Before, everyone would go into the office to work, so meetings were often face-to-face.

Employees use Meet for daily project meetings, monthly catch-ups, and even use them for social events, such as playing multiplayer games with team members. With the seamless integration of Meet with Google Chat , employees can also choose to start a meeting instantly through the Chat platform. This allows impromptu meetings to take place, whether it is a one-on-one or group meeting.

One of the value propositions that Zerodha provides is to empower traders through financial literacy. The marketing team does this by creating blog posts around topics like business valuation and option trading, that aim to educate customers, as well as generate new leads.

Using Docs , employees can view, edit, and share articles that are a work in progress internally in real time, so that they can get immediate feedback and input from each other all in the same document. They also use Docs to share meeting notes where necessary.

“Zerodha's guiding principle is scalability. Our trading platform grew to millions of users with the help of a small tech team. Rather than managing email accounts and servers, we want engineers to focus on building digital solutions that our customers need.”

Improving security and ease of access management with Google SSO

Protecting company and customer data is a top priority for Zerodha. For data protection beyond passwords, the company implemented Google Single Sign-On (SSO) with Two-Factor Authentication (2FA) in compliance with Securities and Exchange Board of India (SEBI) cybersecurity guidelines.

Employees now access enterprise apps with a single login using Google SSO and 2FA. To reduce security risks, employees who leave the company would have their Google account suspended, which in turn automatically disables access to all enterprise applications.

"Employees have access to different systems depending on their roles. SSO with 2FA from Google simplifies user authentication and access from any location and device,” says Sudhakaran. “Due to this architecture, we could move to work from home practically overnight with minimal impact on customers.” Even though pandemic restrictions are easing, Zerodha continues to support work from home as the default working arrangement.

As of November 2021, Zerodha upgraded its 1,000+ employees to the Enterprise plan to enhance security with features such as data loss protection (DLP), which prevents users from sharing sensitive data, and alerts administrators when users log in from unfamiliar devices.

"Employees have access to different systems depending on their roles. SSO with 2FA from Google simplifies user authentication and access from any location and device. Due to this architecture, we could move to work from home practically overnight with minimal impact on customers.”

Scaling up digital onboarding to meet regulatory changes

Each day, Zerodha receives around 25,000 account opening applications, all of which undergo a Know-Your-Customer (KYC) process as required by SEBI in order to minimize fraudulent activities such as identity theft and money laundering. Customers must provide KYC documents such as their Permanent Account Numbers (PAN) cards, to verify their identities. Pre-pandemic, there were about 2,500 accounts opened in a day, currently, we see around 15,000 accounts opened per day.

To simplify the KYC process, Zerodha deployed Vision AI in April 2020, which analyzes scanned copies of PAN cards submitted by users as proof of identity. Using optical character recognition, the system detects text in the images, extracts the PAN names and numbers and compares them to the information the clients themselves provided in their application forms. If their PAN cards don't match the information they provided, users receive an alert and will need to submit the correct documentation in order to open their accounts.

"Google Vision API helps us to validate the PAN card uploaded by the user in real-time, thereby avoiding a manual intervention from the support agent and this helps to significantly reduce the time spent on account opening for new users. By eliminating 80% of the manual process, we continue to improve the customer experience as we grow."

To read this content please select one of the options below:

Please note you do not have access to teaching notes, zerodha - a success story(b).

Publication date: 19 December 2018

Teaching notes

The transparency and consistency of Zerodha provided them mileage and the business model certainly disrupted the Indian brokerage industry. The zero-commission model was successful in attracting consumers and provided tough competition to other brokers. However, competitive advantage coming solely from low pricing may not always be sustainable in the long run. This case explores the sustainability of the business model and looks at alternatives that are important for a long term scale-up of Zerodha.

Garg, A. , Gupta, A. , Maheshwari, G. , Sinha, A. and Sugathan, A. (2018), "Zerodha - A Success Story(B)", . https://doi.org/10.1108/CASE.IIMA.2020.000152

Indian Institute of Management Ahmedabad

Copyright © 2018 by the Indian Institute of Management, Ahmedabad

You do not currently have access to these teaching notes. Teaching notes are available for teaching faculty at subscribing institutions. Teaching notes accompany case studies with suggested learning objectives, classroom methods and potential assignment questions. They support dynamic classroom discussion to help develop student's analytical skills.

Related articles

All feedback is valuable.

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

• ENQUIRE NOW
• FAQ’s-PGDRM
• About Global Risk Management Institute (GRMI)
• Governance Structure of GRMI
• GRMI Partners
• Course Overview
• WHY DO A RISK MANAGEMENT PROGRAMME
• INTERNSHIP AND PLACEMENTS
• STUDENTS’ EXPERIENCE
• WHY STUDY AT GRMI?
• CAREER PROGRESSION AND PLACEMENTS
• Faculty Members
• OTHM QUALIFICATIONS
• WHY DEMAND FOR RISK PROFESSIONALS IS RISING?
• News & Stories
• Life at GRMI
• PG in IT Risk Management
• Add-On Course
• GRMI Courses
• Risk Masterclass: De-Risking Strategy by FICCI and GRMI
• RISK MANAGEMENT DEVELOPMENT PROGRAMME(RMDP)
• BOARD INDEPENDENT DIRECTOR’S RISK PROGRAMMES
• Campus Photos
• Model Risk Code
• Enquire for PGDRM

Research Study on Zerodha

• Research Study

This research study is by Om Prakash Aditya from the batch of July’21-22. He graduated with BA (Eco and English Literature) degree from Delhi University in the year 2018. Prior to opting for a career in Risk Management, he was working as the core team member at Leaders for Tomorrow. According to Aditya, a  career in risk management in India  promises exciting opportunities across all verticals of the industry. It involves a thorough study of the processes, strategies, and objectives of a business and then identifying and mitigating the risks associated with them. The value that it adds to business in terms of risk assurance translates to immense career growth for risk management professionals .

Here’s a case study by Aditya on Zerodha:

By Om Prakash Aditya, PGDRM July’21-22

WHAT IS ZERODHA?

Zerodha Broking Limited is an Indian financial services company offering retail brokerage, currencies and commodities trading, mutual funds, and bonds. Founded in2010 by Nikhil Kamath and Nitin Kamath, the company is headquartered in Bangalore. Zerodha attained a self-assessed valuation of over $1 billion in 10 years. The fastest bootstrapped startup to do so. Zerodha is the largest retail stockbroker in India with a client base of 5 million users. Zerodha contributes more than 15% of daily retail volumes across all Indian Stock Exchanges.

ZERODHA KMPs

Zerodha timeline, before zerodha.

Before Zerodha, there were 3 major hurdles troubling the existing Investors which were preventing the common people from investing in the market.

1. Lack of Knowledge and awareness: The Market operated mostly on Myths and emotions rather than knowledge and strategy.

2. Exorbitant Brokerage Fees: The conventional brokerage firms charged a percentage of total earnings from their client, leading to huge brokerage fees.

3. Very Complex Process of Investing: The process involved a lot of paperwork, and the process was difficult for the common investor to understand.

WHAT ZERODHA CHANGED

1. ZERODHA Varsity: To tackle the lack of awareness in the market, Zerodhalaunched ZerodhaVarsity, which is an extensive collection of the stock market and financial lessons aimed at enabling the common investor to gain enough knowledge about Investing.

2. Discount Brokerage fees: Zerodhalowered down brokerage fees to such an extent that it literally resulted in a revolution. Instead of charging a percentage of profits, Zerodhastarted charging Rs. 20 or 0.03% for all intraday trades, while making equity and direct fund investments completely free of cost.

3. Web-Based Service with a simple UI: To tackle the cumbersome process of investing, they developed a User-Friendly Website and Mobile Application to make it very easy to invest in the market.

BUSINESS MODEL

Low Margin and High Volume:  Zerodhacharges a free to a very minimal amount to the dealers for transactions because of which the trading or exchange volume is generally high. This fee collection of smaller amounts from a larger number of clients leads to good revenue generation for Zerodha.

Total User Base: 5 million active users.

Charges per trade (F&O): Rs 20

Avg no. Of trades per day: 3 million.

Avg operating daily revenue: approx Rs. 60 million.

Low Operating Costs: Zerodhais able to keep its operational costs low because of its completely online structure. It also invests in the continuous development of its UI to keep it user-friendly to attract more users to its platform.

Diversification

Zerodha has diversified its core business to include partners like SmallCase, Streak, and Sensibulland GoldenPi which offer specialized platforms for Investments, Algo and Strategy, Options Trading, and BondsTrading. These platforms help Zerodha in retaining a customer base looking for a more specific platform for their needs.

EQUITY TRADING

Equity trading refers to Trading i.e. buying and selling Company shares on the equity markets with an intent to profit from the purchase/sale of these securities. Equity markets are divided into two segments, Primary market (IPO)and Secondary markets (OTC).

DERIVATIVES TRADING

Derivatives Tradingisa contract between two or more parties where the derivative value is based upon an underlying asset. Traders speculate on the future price of the asset to buy/sell on expected gains from the price difference.

CURRENCY TRADING

Currency trading often referred to as foreign exchange or Forex, is the purchasing and selling of currencies in the foreign exchange marketplace, done with the objective of making profits.

COMMODITIES TRADING

A commodity market is a marketplace for buying, selling, and trading raw materials or primary products. Commodities are often split into two broad categories: hard and soft commodities. Hard commodities include natural resources that must be mined or extracted—such as gold, rubber, and oil, whereas agricultural products or livestock are soft commodities.

MUTUAL FUNDS

A mutual fund is an investment vehicle where many investors pool their money to earn returns on their capital over a period. This corpus of funds is managed by an investment professional known as a fund manager or portfolio manager.

BONDS/G-SECS

Zerodha provides a platform for users to buy and sell Bonds and allows tax-free interest incomes.

Zerodha Coin is an order collection platform that lets you buy mutual funds online, with no commission pass back, directly from asset management companies. Your mutual funds, stocks, currencies, bonds, and more, all in your DEMAT account accessible in one place. Coin charges0%commission direct mutual funds and Govt, corporate, and gold bonds and is India’s largest direct mutual funds platform.

Varsity is an investor education & learning platform from Zerodhawherein investors can learn the nuisance of investing in the stock market. ZerodhaVarsity provides a collection of stock market lessons along with in-depth coverage and illustrations in easy-to-understand language.

SUBSIDIARIES

Rainmatter Foundation

Zerodha set up Rainmatter Foundation, a non-profit foundation that aims to support individuals/organizations/companies working towards problems related to climate change. Rainmatter Foundation has committed $100 million in funds towards the above directions. The Foundation has invested in Terra (an online climate school training climate entrepreneurs on climate science, policy, business, and social justice), and Blue Sky Analytics (which provides real-time and predictive environmental intelligence).

RISKS AND CHALLENGES

Zerodha faces the following Risks and Challenges:

• Technical glitches on the platform because of heavy trading, lead to decreasing customer satisfaction.
• Zerodha is an online platform, it does not have offline support branches so inefficient customer support, and lack of fast customer service is the greatest challenge that this company confronts.
• Delayed Payments to traders and non-adherence to the 48hr payout timeline may lead to customer dissatisfaction and accumulated payables.
• Lack of delivery of valuable advisory reports and analysis of one’s investments and trading activities be it either weekly or quarterly. Most of the big full-service brokers provide research reports.
• Competition Risk as competitors is increasingly offering lower transaction charges which may lead to loss of market share.
• Regulatory changes may affect margins leading to possible losses of revenues. May also affect business ventures.

COMPETITORS

Vision and mission.

Upcoming goals for Zerodhais the aim to add 5-10 million new investors to the Indian stock market with the aid of the platform. “India is very dependent on foreign capital to drive the country. For any country to do well, you need residents to put their money in the market. The money shouldn’t just stay in fixed deposits and real estate. I want to encourage people to educate themselves and put the money in the ecosystem in some way or the other to drive growth,” Nithin concluded.

Zerodha is currently looking to reduce the transaction costs down in its discount brokerage services. The company will be launching low-cost funds in order to attain that target, as per the reports dated September 1, 2021.

Get the full research study here:   Zerodha

This report has been produced by students of Global Risk Management Institute for their own research, classroom discussions and general information purposes only. While care has been taken in gathering the data and preparing the report, the student’s or GRMI does not make any representations or warranties as to its accuracy or completeness and expressly excludes to the maximum extent permitted by law all those that might otherwise be implied. References to the information collected have been given where necessary.

GRMI or its students accepts no responsibility or liability for any loss or damage of any nature occasioned to any person as a result of acting or refraining from acting as a result of, or in reliance on, any statement, fact, figure or expression of opinion or belief contained in this report. This report does not constitute advice of any kind.

More from our PGDRM students:

Research study on Cryptocurrency Exchange Platform

• January 2019
• August 2018
• November 2017
• August 2016

Design and Developed by KodeForest @ All Rights Reserved by KodeForest

• Academic Council
• Accommodation and Connectivity
• Admission Form Devlopment
• Admission Form Submission
• Admissions open-PG in Risk Management (PGDRM)
• Amal Ganguli
• Blog 3 Column
• Board of Directors
• Board of Governors
• Book a call slot
• Chetan Mathur
• Connect with counsellors-PGDRM
• Covid-19 Plan for PGDRM
• CURRICULUM- 1 YEAR PGDRM
• EDUCATION LOAN PARTNER-CREDENC
• ENQUIRE NOW-PGDRM
• Faculty Members In GRMI
• Form submitted-Thank You
• Global Risk Management Institute
• GOVERNANCE STRUCTURE
• GRMI Gallery
• GRMI Reports
• HOW WILL I BE ASSESSED?
• Important Documents – Global Risk Management Institute
• Interested in Jan’21 PGDRM-Meet our Students
• INTERNSHIPS
• Management Team
• Mukesh Butani
• Open Payment Form
• OTHM Qualifications
• P K Mukherjee
• PG in Risk Management (PGDRM) by Global Risk Management Institute (GRMI)
• PGDM Apply Now
• PGDRM Enquiry booking
• PGDRM Fee Structure
• PGDRM-Call with academic counsellor
• Privacy Policy, Cookies Policy and Whistleblower
• PROFESSIONALS SPEAK
• Strategy Board
• STUDENTS’ EXPERIENCE IN GRMI
• Subhashis Nath
• Subodh Bhargava
• SUNIL SETHY
• Suresh Krishnan
• Vijay K. Thadani
• ENQUIRE NOW contact_support
• FAQs-PGDRM supervisor_account

Download Brochure

Talk to counsellor.

Pardon Our Interruption

As you were browsing something about your browser made us think you were a bot. There are a few reasons this might happen:

• You've disabled JavaScript in your web browser.
• You're a power user moving through this website with super-human speed.
• You've disabled cookies in your web browser.
• A third-party browser plugin, such as Ghostery or NoScript, is preventing JavaScript from running. Additional information is available in this support article .

To regain access, please make sure that cookies and JavaScript are enabled before reloading the page.

A Startup that Revolutionised the Indian Broking Industry | Success Story of Zerodha

Every year on January 16, India celebrates National Startup Day.

India now has over 72,000 startups, including 100+ unicorns.

But there are only a handful of bootstrapped startups that went on to become unicorn startups.

So, I decided to share the success story of a startup I admire, a bootstrapped one – Zerodha.

Let’s get to know this startup that made our investing and trading journey smooth like butter (Amul one 😋). 

Table of Contents

About zerodha startup, the kamath brothers, how and when did zerodha start, growth of zerodha, what did zerodha do special, challenges and competition.

Before we move on to the story, check out these amazing startup stats for 2023.

I am pretty sure you must have heard this name a lot over the last 2-3 years.

And I can’t believe I was just 10 when two brothers, Nithin Kamath and Nikhil Kamath, founded Zerodha in 2010. 

Zerodha is a financial services company offering discount brokerage services to retail Indian investors and traders. 

They offer investing and trading services in the stock market, currency and commodities markets, mutual funds, and bonds.

And you won’t believe this, Zerodha is India’s largest brokerage firm in terms of active customers. 

Wow! A bootstrapped startup without spending a penny on advertising has over 1 crore customers with 62 lakh+ active customers.

So, how did this happen?

Well, let’s get the ball rolling.

Nithin Kamath, the founder and CEO of Zerodha, started stock trading at the age of 17. 

Having been introduced to the stock market by his friends, he started trading in penny stocks and made a good amount of money. But in the 2001-02 market crash, he lost it all.

This prompted Nithin to work in a call center at night, while he traded during the day. 

Later, he joined Reliance Money as a sub-broker, and he made a lot of money by handling big client accounts.

Nikhil Kamath, co-founder and CTO of Zerodha, followed in his brother’s footsteps.

Nikhil dropped out of school after the 10th standard to play chess. (Quite a bold decision!)

But at the age of 17, he started working full-time at a call center. He also started stock trading after following his brother and friends.

As Nikhil got more experienced in trading, he started managing the assets of his friends and colleagues. Later, he joined Way2Wealth as a sub-broker.

So, how did these two brothers stumble upon the idea of establishing Zerodha?

Let’s find out.

In 2005, a foreign HNI was impressed with Nithin’s trading portfolio. So, he gave Nithin a ₹25 lakh cheque to manage his Demat account. 

Later, he left his call center job as he started handling more accounts.

But, there was a problem.

There was no single platform where Nithin could switch between different accounts seamlessly.

To tackle this situation, in 2006 he joined Reliance Money as a sub-broker. This allowed him to buy and sell stocks from a single platform. 

But in all this, he realised one thing. There was a huge gap between the commissions charged by the brokers and the actual amount clients got after the transaction.

He also figured out why many young people are not interested in stock trading. The reason was simple – high brokerage charges and a complex trading process.

As Nithin had worked with many brokerage firms, he recognized the challenges faced by traders.

So, after trading for almost a decade, Nithin decided to become a broker. And not just an ordinary broker; he also decided to revolutionise the broking industry.

Then both the brothers quit their jobs and started building Zerodha. But the journey ahead was going to be tough.

The reason behind starting Zerodha was to empower traders by providing a simple-to-use and affordable trading platform.

They simply wanted to eliminate all the hurdles faced by the traders and make their trading experience a walk in the park.

And that’s how they came up with the name Zerodha. ‘Rodha’ is a Sanskrit word for hurdle or barrier. Now, do I need to tell you what Zero means? 😁

When the two words are combined, it means “no hurdles” or “zero hurdles”.

So, how did they remove the obstacles and went on to become the largest brokerage company in India?

Well, here’s the story.

Just before Zerodha was founded, financial markets went through a crisis in 2008-09.

Nithin was looking for VC funding, but VCs were reluctant to fund an online brokerage startup after the market crash. Rather, they were not interested in investing money in any startup.

So, he decided to use his savings to get things moving. And most of the savings went into purchasing exchange refundable deposits and setting up an office space.

Everything was set, and now it was show time.

Zerodha’s operations began on August 15, 2010. And this made them India’s first discount brokerage firm. 

They launched a user-friendly and easy-to-use trading platform using the latest technology.

On top of that, they only charged a flat fee of ₹20 on every trade, irrespective of its size, with zero brokerage fees and an annual maintenance fee of ₹300.

This disrupted the whole brokerage industry. Zerodha did what other traditional brokers couldn’t do.

But they had to tackle some challenges.

One of the major issues was to onboard traders on Zerodha’s platform.

Since they didn’t get any VC funding and had fewer savings, doing a marketing campaign was out of the question.

So, they decided to do community outreach, and Nithin was already part of some online trading communities. This worked in their favour for distributing the early version of Zerodha.

And they didn’t just stop at community building; they also did door-to-door marketing. 

Both of these strategies helped them onboard their first 1,000 customers. (Yay!)

Zerodha did this without spending money on advertising, and they still don’t spend money on it. (No kidding)

In the first year of their operations, they managed to open 3000 Demat accounts. 

Now, you must be thinking, “Just 3000 accounts!”

Yeah. Look, serving in a market like India is not easy. People become suspicious if you do something different from everyone else.

And most traders were reluctant to use Zerodha’s services as they did not offer any research services.

This was their second issue – less credibility.

Unlike full-service brokers, they didn’t offer any stock buy or sell recommendations.

To overcome this, they launched Varsity, a collection of stock market lessons and terms.

Again, they used technology to simplify the stock investing and trading processes. And to retain the customers, they created a superior trading platform. 

After resolving these issues, Zerodha had 17,500 clients investing and trading on Indian stock exchanges. But things were about to go upward.

In 2015, the founders decided to make the delivery of equity investments free of charge. From a fee of ₹20 per order to ZERO fee.

This resulted in the first wave of Zerodha’s growth. In just 5 years, the customer base went from 30,000 accounts (2015) to 14 lakh accounts (2020). 

When all this was happening, in 2019, Zerodha became the market leader by toppling ICICI Securities. 

But this was just the tip of the iceberg. 

In 2020, two things happened.

First, the Covid-19 lockdown fueled the interest of Indians towards stock investing and trading. 

This surge in interest led to massive growth as many new customers opened Demat accounts through Zerodha. 

As a result, their user base surpassed one crore, with over 62 lakh active customers placing millions of orders per day.

Second, after 10 years of operations, Zerodha became a profitable unicorn startup. One of the few profitable unicorns!

And this spectacular growth is also reflected in Zerodha’s revenue and profits.

In FY22, the startup reported ₹4,964 crore in revenue and ₹2,094 crore in profits. And they achieved all this without VC funding and advertising.

So, how did they achieve this?

The Kamath brothers were well aware of the challenges faced by the traders. And they offered a simple solution by building Zerodha’s online trading platform. 

As I said earlier, the founders tried to remove obstacles like high brokerage fees, lack of transparency, and complex trading processes.

And traders were introduced to a smart trading platform at a low cost, which made their trading journey easier. 

Also, Zerodha’s business model was based on low margins and high volume. This model, coupled with zero brokerage, led to good revenue generation.

During this period, they also introduced a flat fee model by challenging the percentage brokerage fee model.

Now, what is the percentage brokerage fee model?

Let me explain this with an example.

Back in the day, traditional brokers would charge brokerage based on your trading value.

Let’s say a broker charged 1% per trade.

So, if your traded value is ₹1,000, your brokerage fee would be ₹10.

Now, imagine your trade value is ₹1 lakh, so your brokerage fee would be ₹1,000.

But Zerodha played a masterstroke by charging only 20 rupees. This was possible because the trade size didn’t affect the operational cost.

And they also showed all the charges transparently by adding a brokerage calculator to their website. This enabled traders and investors to know all the charges with a click.

By doing this, Zerodha earned a reputation among the trading community. And this helped them gain a market share of approximately 17%, closely followed by another discount broker, Upstox (14%).

Let me know if you want a case study on Upstox.

Now, let’s talk about the challenges and competition faced by Zerodha.

No doubt, Zerodha is a pioneer in the brokerage industry, but they also face a few challenges that they need to overcome.

Due to a huge number of transactions, the platform experiences technical glitches. This disrupts the trading and investing experience of the users.

Now, Zerodha doesn’t have any offline branches as most of the operations are performed online. 

So, if any customer faces any difficulty, they need to reach their online customer support team. And it is likely that the customer support team might fail to address the customer’s concerns.

Even the founders admitted this by saying that their customer support is the weakest link of Zerodha. I am sure they will sort this out pretty soon.

And like most discount brokers, they don’t provide any stock advisory services or market-related calls.

All of this may result in customer loss.

Now, let’s look at Zerodha’s competition.

Since the Covid-19 lockdown, the financial services industry has seen a tremendous boom. Now, more people are interested in investing and trading in stocks.

And Zerodha faces stiff competition from many traditional brokers as well as discount brokers. 

The traditional brokers include big players like ICICI Direct, HDFC Securities, Motilal Oswal, and Sharekhan.

And the discount brokers include VC-backed Upstox, Groww, and Angel One. These discount brokers are gaining a strong foothold in the market. 

But the only thing that differentiates Zerodha from other brokers is zero advertising and marketing. 

They built a community that helped them acquire new customers through word-of-mouth marketing.

In which year Zerodha started?

Zerodha started on 15th of August in 2010.

What does Zerodha company do?

 Zerodha is a financial services company based in India that provides a range of investment services, including retail and institutional broking, equity investments, and trading in currencies and commodities.

As a member of NSE, BSE, and MCX, Zerodha offers a reliable and trustworthy platform for investors to manage their portfolios. Most notably, Zerodha is known for its unique offering of brokerage-free equity investments. 

 Is Zerodha startup profitable?

Yes, Zerodha’s FY22 net profit was Rs 2,094 crore, up 87% YoY from Rs 1,122 crore in FY21. Operating revenue surged 82% to Rs 4,963 crore YoY. Notably, Zerodha has not raised any external funding to date.

What is the meaning of Zerodha?

Zerodha, a combination of “Zero” and the Sanskrit word “Rodha,” meaning barrier, reflects the company’s dedication to providing affordable brokerage services to Indian investors while removing obstacles and barriers.

Both the founders, Nithin and Nikhil, used their experience and knowledge to build a product that solved the problems faced by traders.

And solving a burning problem is the main aspect of building a startup. 

Serving in a price-sensitive market like India is not easy. They identified it early on and took some bold decisions that eventually worked in Zerodha’s favour.

Comment down your views on Zerodha’s success.

With that, it’s time to say goodbye! 

1.5+ million PDFs in 25 minutes

14 Feb 2024

At Zerodha, many million users login and use our financial platforms every day. Over the recent months, on an average day, 1.5+ million users have been executing stock and derivative transactions. On a volatile day, this number could easily double. After a trading session concludes and all the number-crunching, tallying, and “backoffice” operations are completed—with file dumps received from stock exchanges and other market infrastructure institutions—stock brokers e-mail a digitally signed PDF report called the contract note to every user who transacted on that particular day. This PDF report offers a comprehensive breakdown of each trade, detailing the total cost associated with each transaction. Its layout, contents, delivery time, and all other aspects are bound by regulations.

Needless to say, this is a complex task to orchestrate at scale. Our legacy system which we had been continually improving over the years, had grown to taking some 8 hours daily, at which point a complete overhaul was required. We typically watch technical debt closely and allow it to grow to a certain acceptable level before addressing it. For this trade-off, we pause development of new features when necessary for the sake of the overall health of our stack.

TL;DR : In this blog post, we describe our journey of rethinking the architecture and building an architecture from scratch which now enables us to process, generate, digitally sign, and e-mail out 1.5+ million PDF contract notes in about 25 minutes, incurring only negligible costs. We self-host all elements of this architecture relying on raw EC2 instances for compute and S3 for ephemeral storage. In addition, the concepts used for orchestration of this particular workflow can now be used for orchestrating many different kinds of distributed jobs within our infrastructure.

• A Python application reads various end-of-the-day CSV dumps coming from exchanges to generate HTML using a Jinja template.
• Chrome, via Puppeteer, converts this HTML into PDF format.
• A Java-based command-line interface (CLI) tool is then used to digitally sign the PDFs.
• Self hosted Postal SMTP servers to send signed PDFs over email.

Before the revamp, the entire process took between 7 to 8 hours to complete, with durations extending even longer on particularly high-volume days. These processes ran on a large single, vertically scaled server that would be spawned for the duration of the work. As the volume of work grew over time, the challenges increased. Additionally, the self-hosted Postal SMTP server started hitting significant performance bottlenecks further affecting the throughput.

The necessity to send these contract notes before the start of the subsequent trading day added significant pressure, as the computation of source data itself is time-consuming and reliant on the timely receipt of various files from exchanges, which themselves experience delays. So, we really had to throw away what we had and think afresh.

It was clear that the only way forward was a horizontally scalable architecture, enabling us to add more capacity as required indefinitely (theoretically at least). Given that stock market volatility can cause the workloads to vary wildly on a daily basis, we had to come up with a system that was not only highly performant and resource efficient, but time bound, thanks to regulations. A conventional FaaS (“Functions as a Service”) offering with cold-start problems and communications overhead, and of course cost overhead per run, would not be ideal. To get large volumes of CPU-bound work done as quickly as possible, going by first principles, for this usecase, it made sense to take a strategy where we would spin up large ephemeral server instances whose CPU cores would be saturated concurrently with work with little time spent on waiting. On completion, the instances could be destroyed immediately, thus gaining optimum resource usage and costing.

In this particular case, the contract note PDF generation workflow is composed of many independent units of work with varying resource requirements. Data processing -> PDF generation -> PDF signing -> e-mailing PDFs, all of which can have independent workers doing their respective jobs and passing the results back into the pool to be picked up by another worker. Since the goal is high throughput and optimum utilisation of available resources, apart from designating CPU pools for different job groups, fine grained resource planning is not necessary. In fact, any sort of granular resource planning is not feasible as the size and scale of individual workloads can vary wildly on a daily basis. One user could have a PDF report with two pages while another could have two thousand pages. So the best bet is to throw all the jobs into a pool of workers running on large ephemeral instances and wait for the last job to be done before tearing down the instances. That one two-thousand page PDF customer on a volatile market day may hold up a core 100 times longer than another customer, but because we spawn a large number of cores and burn through the jobs concurrently, it evens out. This approach worked out really well for us.

Architecture

To begin with, we rewrote the Python worker process from scratch as concurrent Go programs, immediately availing performance and resource usage benefits by orders of magnitude. For distributing job processing between various worker instances, after evaluating existing options, we ended up writing a lightweight, pluggable Go messaging and job management library— Tasqueue .

The first sequence in our chained jobs workflow begins with a generator worker processing various CSV files to create templates for PDF files. This is then pushed into the queue as a job for the next worker, say the PDF generator, to pick up. Various different kinds of workers upon picking up their designated jobs, retrieve the relevant file from S3, process it, and then upload the output back to S3. So, for a user’s contract note PDF to be delivered to them via e-mail, their data passes through four workers (process data -> generate PDF -> sign PDF -> e-mail PDF), where each worker after doing its job, dumps the resultant file to S3 for the next worker to pick up.

The global job states are stored in a Redis instance. It serves a dual role in this architecture: as a backend broker facilitating the distribution of jobs among workers and as a storage medium for the status of each job. By querying Redis, we can track the number of jobs processed or identify any failures. For jobs that fail, targeted retries are initiated for users whose jobs previously failed or were not processed.

Generating PDFs

Initially, PDFs were generated from HTML using Puppeteer, which involved spawning headless instances of Chrome. This of course grew to be slow and quite resource-intensive as our volume grew.

After several experiments including benchmarking PDF generation with complex layouts using native libraries in different programming languages, we had a breakthrough using LaTeX instead of HTML and generating PDFs using pdflatex . By converting our HTML templates into TEX formats and utilizing pdflatex for PDF generation, we observed a 10X increase in speed compared to the original Puppeteer way of generating. Moreover, pdflatex required significantly fewer resources, making it a much leaner solution for our PDF generation needs.

Problems with LaTeX

We rewrote our PDF generation pipeline to use pdflatex and ran this in production for a couple of months. While we were happy with the significant performance gains with LaTeX, this process had its own challenges:

• Memory Constraints with pdflatex : For some of our prolific users, the PDF contract note extends to thousand pages as mentioned earlier. Generating such large documents leads to significantly higher memory usage. pdflatex lacks support for dynamic memory allocation, and despite attempts to tweak its parameters, we continued to face limitations in terms of memory usage.
• Switch to lualatex : In search of a better solution, we transitioned to lualatex, another TeX to PDF converter known for its ability to dynamically allocate more memory. lualatex resolved the memory issue to a considerable extent for large reports. However, while rendering such large tables, it sometimes broke and produced indecipherable stack traces which were very challenging to understand and debug.
• Docker image size : We were using Docker images for these tools, but they were quite large thanks to the required TeX libraries for both pdflatex and lualatex, along with their various 3rd party packages. Even with caching layers, the large size of these images resulted in delayed startup time for our instances on a fresh boot up before our workers could run.

In our search for a better solution, we stumbled upon Typst and began to evaluate it as a potential replacement for LaTeX.

Typst - a modern typesetting system

Typst is a single binary application written in Rust that offers several advantages over LaTeX.

• Ease of Use : Typst offers a more user-friendly developer experience compared to LaTeX, offering a simpler and consistent styling experience without the need for numerous third-party packages.
• Error Handling : It provides better error messages making debugging any such issues related to bad input data significantly easier.
• Performance : In our benchmarks, Typst performed be 2 to 3 times faster than LaTeX when compiling small files to PDF. For larger documents, such as those with tables extending over thousands of pages, Typst dramatically outperforms lualatex. A 2000-page contract note takes approximately 1 minute to compile with Typst, in stark contrast to lualatex’s 18 minutes.
• Reduced docker image size : Since Typst is a small statically linked binary, the Docker image size reduced significantly compared to bundling lualatex/pdflatex, improving the startup time of our worker servers.

Digitally signing PDFs

Regulations mandate that contract note PDFs should be digitally signed and encrypted. At the time of writing this, we have not found any performance-focused FOSS libraries that can batch sign PDFs, thanks to the immense complexities in PDF digital signatures. We ended up writing a small HTTP wrapper on top of the Java OpenPDF library, enabling a single boot JVM which can then handle signing requests concurrently. We deploy this server as a sidecar alongside each of our signer workers.

Generating and storing files at high-throughput

The distributed contract note generation workflow processes client and transaction data, generating PDF files typically ranging from 100kb to 200kb per user, where the PDFs can also run into MBs for certain users. On an average day throughout the workflow, this comes up to some 7 million ephemeral files that are generated and accessed throughout the workflow (Typst markup files, PDFs, signed PDFs etc.). Each job’s execution is distributed across an arbitrary number of EC2 instances and requires access to temporary input data from the preceding process. Shared storage allows each process to write its output to a common storage area, enabling subsequent processes to retrieve these files, eliminating the need to transfer files back and forth between job workers over the network.

We evaluated AWS’ EFS (Elastic File System) in two different modes : General Purpose mode and Max I/O mode. Initially, our tests revealed limited throughput as our benchmark data was relatively small. Without specified throughput provisioning, EFS imposes a throughput limit based on data size. Consequently, we adjusted our benchmark setup and set the provisioned throughput to 512Mb/s.

Our benchmark involved concurrently reading and writing 10,000 files, each sized between 100kb and 200kb.

• In General Purpose mode, we reached the EFS file operation limit (35k ops/sec, with reads counting as 1 and writes as 5) and experienced latencies resulting in 4-5 seconds to write these files to EFS.
• Performance deteriorated in Max I/O mode, taking 17-18 seconds due to increased latency. We dismissed Max I/O mode from our considerations due to its high latency.

Given the large number of small files, EFS seemed wholly unsuitable for our purpose. For comparison, performing the same task with EBS took approximately 400 milliseconds.

We revised our benchmark setup and experimented with storing the files on S3, which took around 4-5 seconds for a similar number of files. Additionally, we considered the cost differences between EFS and S3. With 1TB of storage and 512Mb/s provisioned throughput, S3’s pricing was significantly lower. Consequently, we opted to store our files on S3 rather than EFS, given its cost-effectiveness and the operational limitations of EFS.

We also consulted with the AWS Storage team, who recommended exploring FSx as an alternative. FSx offers various file storage solutions, particularly FSx for Lustre , which is commonly used in HPC environments. However, since FSx was complicated to set up and unavailable in the ap-south-1 region during our experimentation—coupled with our operations being restricted to this region—we opted for S3 for its ease of management.

We rewrote our storage interface to use S3 (using the zero-dependency lightweight simples3 library which we developed in the past), but hit another challenge this time: S3 Rate Limits. S3’s distributed architecture imposes request rate limits to ensure fair resource distribution among users.

Here are the specifics of the limits:

• PUT/COPY/POST/DELETE Requests: Up to 3,500 requests per second per prefix.
• GET/HEAD Requests: Up to 5,500 requests per second per prefix.

When these limits are exceeded, S3 returns 503 Slow Down errors. While these errors can be retried, the sporadic and bursty nature of our workload meant that we frequently encountered rate limits, even with a retry strategy of 10 attempts. In a trial run, we processed approximately 1.61 million requests within a 5-minute span, averaging around 5.3k requests per second, with an error rate of about 0.13%. According to the AWS documentation , to address this challenge, the bucket can be organized using unique prefixes to distribute the load.

Initially, for each customer’s contract note, we generated a unique ksuid . These are not only sortable but also share a common prefix. Eg:

After consulting with our AWS support team to understand why we were encountering rate limitations despite creating unique prefixes, we discovered that the bulk of our requests were being directed to the same partition, 2CTgQ .

Each new partition increases the allowable request rate by an additional 3.5k for non-GET requests and 5.5k for GET requests per second. This scaling process continues until the prefix structure cannot accommodate further expansion. If we send requests across a completely different prefix next time, for example, bucket/3BRgJLV/ , the same process will start for this prefix as well, while the partition that was created across 2CTgQ would be running cold. When a specific partition runs cold, the partitioning process would need to start over when request rates are high again. Thus, to fully benefit from S3’s partitioning capabilities, it is crucial to consistently utilize prefixes that have already been partitioned to achieve higher transactions per second (TPS). Armed with this knowledge from the AWS support team, we opted to revise our partitioning schema, moving away from the use of ksuid .

We ended up with a simple, fixed schema: {0-9}-{pdfs, signed_pdfs, contract_notes} .

With this, all API requests are now evenly distributed across these 10 fixed partition keys, effectively multiplying our original rate limits per key by tenfold. Furthermore, we requested AWS to have the partitions in our S3 bucket be pre-warmed. Given the nature of our operations—a batch job with a high throughput and bursty request rates—we anticipated potential cold start issues if a partition key weren’t already allocated to handle a high volume of requests. We specifically want to thank the AWS S3 team here for the deep-dive.

Orchestrating with Nomad

We chose Nomad as the foundation for our orchestration framework as all our deployments at Zerodha already use Nomad. We actually like it and understand it well, and it is far easier to run and use as an orchestrator than something like Kubernetes.

We use the Terraform module which we developed to provision a fresh Nomad cluster (including server and client nodes) every midnight before running the batch job process. Here’s an example of how we invoke a Nomad client module:

This module facilitates the setup of all essential infrastructure including launch templates, Auto Scaling Groups (ASGs), IAM roles, and security groups, required to deploy the ephemeral fleet of nodes that we use to process the bulk jobs. We utilize multiple ASGs, each handling load for different groups of servers handling different jobs such as signing PDFs, sending emails, and generating PDFs.

For the admin management and execution, we rely on a small control instance server that is equipped with Rundeck for orchestrating the entire workflow, from infrastructure creation to job execution and completion.

The control instance serves as the central orchestrator, executing scripts across three key phases of the batch job:

Initialization : The process starts by running a Terraform pipeline to launch both Nomad server and client nodes. The client nodes are where our job workers will run. Upon cluster readiness, the script verifies that all nodes are eligible for scheduling jobs by the Nomad scheduler before proceeding.

Job Deployment : Once the nodes are up, Nomad deploys our job worker programs (Go binaries pulled from S3 for the most part) on designated nodes, where they listen for incoming jobs, enabled by the distributed task management library.

Nomad offers various scheduling strategies suited for diverse workload types. For our needs, system jobs proved particularly valuable. Like Kubernetes’ DaemonSets, system jobs ensure that one instance of the job runs on each node, guaranteed by the Nomad scheduler.

Beyond utilizing system jobs, we also use the constraints stanza for further control over worker placement. Through our Terraform module, we pre-populate EC2 tags within the Nomad client configuration, making these tags accessible within Nomad’s client meta. These tags then serve as the basis for defining constraints, enabling dynamic job scheduling and node assignment based on specific attributes.

Using the nomad_client EC2 tag, we determine the role of each client and deploy the corresponding program, which for the most part, are our worker programs written in Go. In the example above, you can see separate ASGs for signer and pdf-creator tasks. This enables Nomad to ensure they run on distinct sets of nodes for optimal resource utilization.

PDF generation requires significantly more resources than signing tasks, so we use separate ASGs for these processes to scale them independently of other jobs.

Once jobs are initiated, we additionally stream the job statuses to the RunDeck UI from the Redis instance that maintains the global state of all distributed jobs, in case an admin wants to peek.

The Rundeck control server runs a Python script to extract job status data from Redis:

For the contract note generation job, we spawn about 40 instances in total currently, a mix of c6a.8xlarge , c6a.2xlarge , and c6a.4xlarge .

Post-execution teardown

Upon the completion of all queued jobs—in this example, the computation, generation, signing, and e-mailing of 1.5+ million PDFs—we initiate the teardown process. A program that monitors the successful job count in the Redis state store executes this by simply invoking a terraform teardown. This involves resetting the ASG counts to zero, draining existing nodes, halting Nomad jobs, and shutting down the Nomad server itself.

In this specific example, the entire operation, end-to-end, finishes in 25 minutes. The cost incurred is unsurprisingly, negligible.

E-mailing PDFs at high-throughput

As PDF signing workers sign PDFs and hand them over, they are instantly queued for being e-mailed by the e-mailing workers. We use a self-hosted auto-scaling Haraka SMTP server cluster and maintain concurrent connection pools from the e-email workers with the smtppool library we wrote to push out e-mails at high throughput.

We transitioned from Postal that we used for many years to Haraka for significant performance benefits—a change that merits its own post separately. Postal’s resource usage was intense and it was not horizontally scalable, and unfortunately grew into an unfixable bottleneck. However, with our move to Haraka which can be easily horizontally scaled, we are no longer limited in capacity pushing out e-mails from our SMTP cluster to target mail servers over the internet. It is important to note that IP reputation matters when self-hosting SMTP servers and we have grown and maintained this over almost a decade—mainly by never sending marketing e-mails and definitely not spam!.

So, that’s it. We are very pleased by the throughput that we have achieved with the new architecture, primarily with the breakthroughs that are Typst and Haraka, and all the orchestration headaches that Nomad trivially handles for us. This is not one big software system, but more of a conceptual collection of small scripts and programs that orchestrate workflows for our specific bulk jobs. We are also happy that this also resulted in the creation and open sourcing of multiple projects.

We plan on moving all our time consuming bulk jobs—of which there are plenty in our industry—to follow the same framework. In addition, we are in the process of adding metrics to the workers so that we can track the global workflow states in realtime and adjust resources as necessary.

As the architecture is simple and rooted in common sense practices conceptually, the fundamental building blocks can be swapped out entirely if required (RunDeck, Nomad, S3 etc.). We are confident that this will do its job and scale well for a long time, and there is room for further optimisation. Needless to say, spawning a few large barebones instances, running highly concurrent business logic saturating all cores throughout and then winding down as quickly as possible, the cost incurred is negligible, and any potential resource wastage is minimal, if not nil.