Some people in life you don’t forget. For me, one of these people is Shawn Dorsch, president of Clear Market Holdings, an electronic trading company. I first met Shawn at Art’s Barbecue in Charlotte, NC, after he agreed to be a mentor to our company.

Shawn has the impressive skill of switching contexts effortlessly. One moment, he would be explaining to us the intricacies of the stock market. The next, he would be on the phone with an investor in Tokyo, rattling off Japanese. Shawn was generous with us, sharing many lessons learned from his more than 20 years in finance. And he was clear about one piece of advice  —  that we read Michael Lewis’ Flash Boys. He even sent us a hard copy of the book for us to read.

Flash Boys opened my eyes to the complexity of stock trading and how the system could be gamed. Obtuse terms like “dark pools” and “high-frequency trading” took on a life of their own. And though the book deals with the arbitrage created by high-frequency trading from 2007-2014, it is about more than a specific time period. As the author points out:

“The 1987 stock market crash set into motion a process  —  weak at first, stronger over the years  —  that has ended with computers entirely replacing the people.” 

Lewis takes us on a journey through the evolution of stock exchanges, showing how changes in technology had wide ramifications, sometimes enabling bad actors. The effects of these changes are still being felt today, though they take different forms. Today’s “high-frequency traders” deal in terms such as AI and big data instead of trading speed, as reported by The Financial Times:

“In no time at all, the lucrative profits to be had from HFT began to encounter limits (fast-as-light trading, regulation, et cetera), bringing forward a new equilibrium. Now, tellingly, with their profits under threat, HFT companies are looking to invest in new techniques to recapture their edge. Much of this is focused on big data analytics, and of course AI and machine learning.”

The Start of Digitization  —  1987

The stock market crash of 1987 set into motion a trend of digitization. At the time, stock brokers would take orders from their clients over the phone. Afraid of big losses, many brokers refused to pick up the phone. This led the SEC to push for electronic trading. If an electronic router was handling customer orders, then humans could no longer sabotage them. That, at least, was the reasoning.

This, combined with the privatization of stock exchanges, created the perfect scenario for arbitrage. High-frequency trading firms would pay big money to have faster connections to the exchanges, then exploit price differences at different exchanges. Earning a penny here and there on each trade, they pocketed billions.

The Privatization of Stock Exchanges —2005-2007

In 2005, the US government passed regulations that forced stock exchanges like NASDAQ and the New York Stock Exchange to become publicly traded companies. The driving force of the change was a complaint that cronyism had taken over. Privatization would increase competition and lower costs. Again, that was the reasoning. 

This change had good and bad effects. Many new exchanges opened up, such as BATS and Direct Edge. This increased competition and lowered the exchanges’ profit margin. However, the drop in profit made the exchanges vulnerable to the demands of high-frequency firms. These firms paid the exchanges for “colocation” — close proximity to the exchange’s servers. This gave them the extra speed to take advantage of other investors. 

A positive effect of the change was a reduction in trading costs. This is an area that has improved drastically over the years. Take this chart from a study on trading costs over the years, which shows commissions in 1968:

Because of increased competition, stock exchanges could no longer enforce large margins on trades. Commissions for buying and selling stock dropped significantly. Today, commissions for buys and sells are fractions on the penny, or roughly $0.0001 per trade.

This paved the way for brokerage firms to reduce costs. In the past year alone, firms like Fidelity and Charles Schwab have reduced commissions from $8-10 down to $5 per trade. This change also enabled firms like Robinhood to offer commission-free stock trading.

The Opening of Custodial Services — 2012

In 2012, the company APEX Clearing became the first company to offer custodial services to other brokers. Prior to this, it was a capital-intensive process to start a brokerage and service clients. This is because the broker had to own the “full stack” of brokerage services, which included custody and clearing of securities. 

When APEX offered a suite of APIs to handle the custodial process digitally, it opened the path for a new breed of brokerage platforms  —  the robo-advisors. These companies, including Wealthfront and Betterment, leveraged the massive scale of software to offer low-cost investment guidance and services. 

Many of the newer brokerage products such as Stash Invest and Acorns rely on APEX. Even Robinhood had to wait until 2013 to launch a commission-free brokerage product, because it too relies on APEX.

The Pursuit of AI and Financial Intelligence  —  2015

Hedge funds and investment firms have now turned their sights towards AI and machine learning for a competitive edge. Many consumer apps, such as Foursquare, are now in the business of selling big data. Firms are on the lookout for anything that can provide “signal.” 

As hedge funds seek out proprietary data sources to power their algorithms, larger firms are placing big bets on technology, too. 44% of the new job openings at Goldman Sachs are in technology, more than in operations, securities, and investment management, as shown here.

There are also platforms online such as Quantopian, where programmers can share their trading algorithms and compete for prizes. Hedge funds such as Renaissance and Two Sigma have invested heavily in recruiting top talent in data science and machine learning. Although technologists were coveted from the days of high-frequency trading, the demand has only intensified since.

Conclusion

Flash Boys rightly pointed out that stock trading is headed towards complete digitization. We have already witnessed the growing pains of that transformation, ranging from the expansion of stock exchanges to the rise and fall of high-frequency trading. 

Will innovations like machine learning and AI shape the market in ways we cannot predict? How will they affect exchanges, traders, and the corporations whose shares are traded by their algorithms? As in all other instances, they will most likely have both positive and negative impacts. After all, only one thing remains certain in the markets  —  uncertainty.

Tom Goldenberg is CTO & co-founder of Commandiv, a stock-trading platform with recommendations. He was designated as one of LinkedIn's Top Ten Voices in Technology in 2016 and has appeared on the BBC and numerous podcasts to discuss software engineering and entrepreneurship. He specializes in mobile development, machine learning and software engineering.