Why Is It So Hard to Beat the Market?

March 15, 2024
Estimated Reading Time: 7 Minutes

For many investors, beating the market is the ultimate goal. It's easy to see why. Who wouldn't want returns that rise above the benchmarks?

In practice, it's hard to do consistently. Legendary investors like Warren Buffett and Peter Lynch are famous partly because their long records of outperformance are so rare.

In this article, we'll explain what "the market" means, what it takes to beat it, and why it's so difficult. We'll also show a more useful way to measure your portfolio's performance.

What is "the market"?

When people talk about "the market," they usually mean the S&P 500. The index includes about 500 large U.S. companies across sectors such as technology, health care, financials, consumer goods, and industrials. A committee selects them based on size, trading volume, profitability, and other criteria. Together they represent roughly 80% of the value of U.S. publicly traded stocks.

Because it's so widely followed, investors often compare their portfolios to the S&P 500. But it only measures one part of the investment world: large U.S. companies. Other parts of the market have their own benchmarks:

  • Mid-sized U.S. companies: S&P MidCap 400 Index
  • Small U.S. companies: Russell 2000 Index, which tracks about 2,000 smaller U.S. stocks
  • Very small U.S. companies: CRSP US Micro Cap Index
  • Developed international stocks: MSCI EAFE Index, covering developed markets outside the U.S. and Canada
  • Emerging market stocks: MSCI Emerging Markets Index
  • U.S. investment-grade bonds: Bloomberg U.S. Aggregate Bond Index, which includes Treasury, government agency, corporate, mortgage-backed, and asset-backed bonds

What does it mean to "beat" the market?

Beating the market means earning a higher return than a chosen benchmark over the same period. In effect, it means doing better than you would have by simply holding a low-cost index fund that tracks that benchmark.

For the comparison to be fair, the benchmark should match what you actually own. Comparing a portfolio of small-company stocks to the S&P 500, or a bond portfolio to a stock index, tells you little about how well the portfolio was managed.

Why is it so hard?

A few forces work against investors trying to outperform:

  • Competition: Millions of professional and individual investors are analyzing the same companies. New information tends to be reflected in prices quickly, which makes consistent mispricings hard to find.
  • Costs: Fees, trading costs, and taxes reduce returns. An index's return doesn't include these costs, so an investor has to beat the index by enough to cover them.
  • Concentration: A small number of very large companies has driven much of the S&P 500's recent performance. Portfolios that hold less of those companies can trail the index even if their other holdings do well.
  • Consistency: Outperforming in one year is very different from doing it year after year. Managers who outperform in one period often don't repeat it.

The data reflects this. S&P Dow Jones Indices' SPIVA research has consistently found that most actively managed U.S. large-cap funds trail the S&P 500 over long periods.

What if my portfolio doesn't beat the market?

A successful investment experience doesn't require beating the S&P 500. What matters more is whether your portfolio is helping you meet your long-term goals at a level of risk you're comfortable with.

Trying to make your whole portfolio beat an all-stock index usually means holding mostly or entirely stocks. That may mean taking on more risk than your goals call for:

  • Higher volatility: Stock prices move with market sentiment, economic conditions, and company news. An all-stock portfolio is likely to see larger swings and deeper losses in downturns.
  • Less diversification: A portfolio without bonds, cash, or other asset classes has fewer sources of return and may go through longer stretches of losses or underperformance.
  • Risk of permanent loss: Concentrated positions in individual companies can lose value permanently if those companies run into serious trouble.
  • Emotional decisions: Steep declines can make it tempting to sell at the wrong time. In early 2020, the S&P 500 fell about a third in roughly five weeks. In 2022, it fell about a quarter from January to October.

For most investors, a mix of stocks and bonds, tailored to their situation, is a better fit than an all-stock portfolio. The real test of a portfolio is whether it helps you reach your financial goals, not whether it beats an index in any given year.

How should I measure my portfolio's performance?

Instead of comparing your whole portfolio to the S&P 500, compare it to benchmarks that reflect what you actually own. There are two useful ways to do this.

Compare each part to its own benchmark. Look at each asset class in your portfolio separately. For example, if your portfolio is 60% large U.S. stocks and 40% bonds, compare the stock portion to the S&P 500 and the bond portion to the Bloomberg U.S. Aggregate Bond Index.

Compare the whole portfolio to a blended benchmark. A blended benchmark combines the relevant indexes in the same proportions as your portfolio. For the example above, it would be 60% S&P 500 and 40% Bloomberg U.S. Aggregate. Comparing your total return to this blend gives a much fairer picture than comparing it to any single index.

A few other points help keep the comparison fair:

  • Use the same time period for your portfolio and the benchmark.
  • Keep in mind that your returns reflect fees and costs, while index returns don't.
  • Look at risk as well as return. A portfolio that trails a benchmark slightly while taking on much less risk may be doing exactly what it was designed to do.

Ensure your portfolio is helping you achieve your financial goals.

It's essential for investors to have a clear understanding of their objectives, risk tolerance, and investment strategy, and to focus on achieving their long-term financial goals rather than chasing short-term market outperformance.

To learn more about the investment strategies we use for our clients, we invite you to contact one of our Wealth Advisors. Or, if you'd like to receive a complimentary Portfolio X-Ray that examines your portfolio against the relevant benchmarks, you can contact us here.


Important Disclosure Information

Past performance may not be indicative of future results. Different types of investments involve varying degrees of risk, and there can be no assurance that the future performance of any specific investment, investment strategy, or product (including the investments and/or investment strategies recommended or undertaken by Chicago Partners Investment Group LLC (“CP”), or any non-investment related content, made reference to directly or indirectly in this commentary will be profitable, equal any corresponding indicated historical performance level(s), be suitable for your portfolio or individual situation, or prove successful. Due to various factors, including changing market conditions and/or applicable laws, the content may no longer be reflective of current opinions or positions. Moreover, you should not assume that any discussion or information contained in this commentary serves as the receipt of, or as a substitute for, personalized investment advice from CP. Please remember to contact CP, in writing, if there are any changes in your personal/financial situation or investment objectives for the purpose of reviewing/evaluating/revising our previous recommendations and/or services, or if you would like to impose, add, or to modify any reasonable restrictions to our investment advisory services. CP is neither a law firm nor a certified public accounting firm and no portion of the commentary content should be construed as legal or accounting advice. A copy of the CP’s current written disclosure Brochure discussing our advisory services and fees continues to remain available upon request.