Global Financial Data has calculated a 100-share index that is more comprehensive than existing long-term indices of the United States stock market. GFD has collected historical data on U.S. shares from 1791 until 2017 and is using this database to calculate a more representative index of U.S. shares than is currently available.
The GFD-100 Index is superior to existing indices for three reasons. First, the index is capitalization weighted from its beginning in 1791 until 2017. Second, the selection of stocks is based upon all shares that were traded on U.S. exchanges and over-the-counter. The selection of shares is not limited to the New York Stock Exchange (NYSE), but includes shares from regional exchanges and over-the-counter markets. Third, the index includes finance stocks throughout its history. Finance shares were excluded from the S&P 500 Composite until July 1, 1976 when the composition of the S&P 500 was changed from 425 Industrials, 60 Utilities and 15 Rail shares (adopted in March 1957) to 400 Industrials, 40 Utilities, 20 Transportation and 40 Finance stocks.
Few people realize that bank and insurance stocks were excluded from any major United States stock index until 1976. Although bank and finance stocks mostly traded over-the-counter before the 1970s, finance stocks represented between 10% and 20% of the overall capitalization of the stock market before 1976. Bank and insurance companies were excluded because consistent price quotes were not available and finance stocks were not as liquid as the shares traded on exchanges. The exclusion of over-the-counter shares was not limited to banks and insurance companies. Standard Oil and its subsidiaries also traded over-the-counter and were excluded from stock market indices until they moved onto the New York Stock Exchange. Nevertheless, people did invest in bank and insurance stocks, and in the Standard Oil companies, and their exclusion creates biases in the historical calculations of stock market indices.
In addition to the GFD-100, GFD has calculated separate indices for Transportation and Finance shares. It is important to include bank and insurance shares in making international comparisons of stock market performance. Outside of the United States, finance shares traded on domestic exchanges and are included in domestic indices. It is only the United States that excluded finance shares from their indices. The GFD-100 index makes an attempt to correct this oversight.
GFD’s 100-share Index
Existing calculations of long-term stock market returns in the United States are based upon four primary sources: Smith and Cole (1803-1862), Macaulay (1857-1871) Cowles (1871-1928) and Standard and Poor’s (1928-2017). Unfortunately, these indices have major flaws in them that create biases that researchers have tolerated until now because no one has ever collected historical data on U.S. share prices, corporate actions (dividends and splits) and shares outstanding so accurate price and return indices could be calculated. The current S&P Composite includes data from the Cowles Indices from 1871 until 1927, the 90-share daily S&P Index from January 1928 until February 1957, the 500-share composite that excludes finance stocks until July 1976 and the all-sector 500-share index since July 1, 1976.
Global Financial Data has collected data on all major United States exchanges going back to 1791 as well as data on over-the-counter shares since 1865. The data includes not only share price data, but information on corporate actions (dividends and splits) as well as shares outstanding. These data sources enable us to calculate cap-weighted price and return indices for the United States that accurately reflect what an investment in the 100 largest stocks each year would have produced for investors.
GFD set up criteria for determining which stocks are included or excluded from its indices. GFD has followed these rules for inclusion:
1) There have to be at least 9 observations per year for each stock and there have to be at least two price changes during each year to warrant inclusion. Otherwise, the stock was excluded due to illiquidity.
2) Dividend data have to be available for each stock in order that both price and return indices can be calculated. A stock may not have paid a dividend during a particular year, but to be included, we had to know the company had passed on paying a dividend during that year. Any shares for which we were unable to find dividend information was excluded.
3) There had to be share outstanding information available so the stock could be included in a capitalization-weighted index
4) Only non-assessable shares were included. A stock was excluded if it imposed assessments on shareholders. Shares that were liquidating assets and were paying liquidating dividends were excluded as well.
The index includes all shares that met these criteria from 1791 to 1824, the top 50 shares by capitalization are used from 1825 to1850 and the top 100 shares by capitalization from 1851 to 2017. Although GFD plans to calculate indices that include more than 100 shares in the future, we decided that because the index was capitalization-weighted, a 100-share index would track a 500-share index very closely. The graph below compares the performance of the S&P 100 (green line) and the S&P 500 (black line) between 1976 and 2018. The performance of the two is very similar. The S&P 500 did outperform the S&P 100 by 0.5% per annum, but the returns are close enough to show that a strong correlation exists between the top 100 shares and the top 500 shares. In 2017, the top 100 shares in the S&P 500 represented about 60% of the total capitalization of the S&P 500.
To choose which companies to include, shares were weighted by capitalization during the first month of each year and included if they were among the 100 largest shares in the United States. It was assumed that the stocks were held for the rest of the year, and in January of the next year, the same selection methodology was used to choose which shares to hold for the coming year. Each year, the list of the 100 largest companies was recalculated and a new list of stocks was introduced. For continuity purposes, if a stock missed a year, i.e. the stock was in the top 100 in 1914 and 1916, but not in 1915, the stock was included in the index in 1915 even though this put over 100 stocks in the index.
Initially, there were less than 100 shares in the index because of a lack of companies that met the criteria outlined above. There were only 3 companies in 1791, 7 in 1800, 18 in 1810, 25 in 1820 and 50 in 1825. The index included 50 members through 1850 and beginning in 1851, there were 100 shares in the index.
The chart below shows the sectoral composition of the GFD-100 index from 1800 until 2017 to cover the period before the introduction of the Cowles indices in 1871. However, the Smith and Cole indices have some major biases in them. The indices use a very limited number of companies, excluding, for example, the first and second Bank of the United States, which represented from 50% to 80% of the overall capitalization of the United States stock market. Consequently, Smith and Cole exclude over half of the U.S. stock market in their indices.
As can be seen, finance stocks (banks and insurance companies) represented over 95% of the capitalization until 1825. After 1825, the transportation sector grew, first with canals and then with railroads. By the 1840s, transports represented over half of the total capitalization, growing to 80% by the 1870s, then shrinking as industrials began to dominate the U.S. stock market.
By 1900, industrial stocks represented about 40% of total capitalization with Energy stocks, primarily Standard Oil and its subsidiaries, representing a growing portion of total capitalization. It should be noted that Standard Oil of New Jersey and California (ExxonMobil and Chevron) weren’t included in the Cowles indices until 1918 when Standard Oil listed on the NYSE. By 1918, Standard Oil had grown tenfold in size to become the largest company in the world, yet was not included in any indices because it traded over-the-counter. And as we stated previously, no finance stocks were included in either the Cowles indices or the S&P indices until 1976. Therefore, we feel that the composition of stocks in the GFD indices are more representative of the stocks that were available to investors in the United States than any of the existing indices of United States stocks.
To show how the GFD-100 index differs from existing indices of U.S. stocks we will compare the components and performance of the GFD-100 index and other indices.
Survivorship Bias in the Smith and Cole Indices
Currently, no index for U.S. stocks exists prior to 1802 when the Smith and Cole indices begin. Therefore, the GFD-100 index covers data that is currently unavailable from existing indices. The graph below compares the Cole and Smith indices (USCFALLM) with the GFD-100 index (GFUS100MPM) from 1802 until 1845 when Smith and Cole’s Bank and Insurance index was discontinued. As can be seen, the Smith and Cole index outperforms the GFD-100 index; however, in analyzing how the Smith and Cole indices were put together, we discovered that the superior performance was caused by the survivorship bias that Smith and Cole built into their indices.
In Walter B. Smith and Arthur H. Cole, Fluctuations in American Business, 1790-1860, Cambridge: Harvard Univ. Press, 1935, they provided calculations of stock market indices for Banks (1802-1820), Banks and Insurance (1815-1845) and Railroad (1834-1862) shares which have been used to cover the period before the introduction of the Cowles indices in 1871. However, the Smith and Cole indices have some major biases in them. The indices use a very limited number of companies, excluding, for example, the first and second Bank of the United States, which represented from 50% to 80% of the overall capitalization of the United States stock market. Consequently, Smith and Cole exclude over half of the U.S. stock market in their indices.
For the dataset of bank stocks from 1802 until 1820, Smith and Cole collected data on 21 bank stocks, 9 insurance shares, 5 bridge and turnpike shares and 3 miscellaneous companies. Smith and Cole based their index on 7 bank stocks which existed during the entire 19-year period from 1802 until 1820, creating a survivorship bias. Banks such as the first Bank of the United States were excluded because they did not survive until 1820. Since Smith and Cole collected no data on shares outstanding, the index was calculated as a geometric mean of the underlying shares. The index is not cap-weighted even though it included only 7 banks. Because Smith and Cole collected no data on dividends, no return index was calculated. Scholars who have attempted to calculate total returns have calculated a dividend yield based upon the yield on government stocks. Jeremy Siegel in his book Stocks for the Long Run, uses a 6.4% dividend between 1802 and 1870.
Smith and Cole collected data on 17 bank stocks, 5 canal stocks, 3 gas-light stocks, 19 insurance stocks, 27 rail, 6 miscellaneous and 17 Boston bank shares between 1815 and 1845. From these shares, they chose 6 bank shares and 1 insurance company for the Bank and Insurance index. Again, only stocks that survived the 30-year stretch between 1815 and 1845 were included, once again creating a survivorship bias.
Finally, Smith and Cole calculated a railroad index that went from 1834 to 1862. Smith and Cole removed shares that either changed little or were erratic in their behavior. They broke up the data into 8 shares from1834 to 1853 and in several cases, because the data series were incomplete, they “spliced” together some of the series even though they admitted this was a statistical device of “questionable value.” Smith and Cole calculated two rail indices of 8 and 10 shares from 1834 to 1853 and a composite of 18 shares from 1853 to 1862. Finally, they calculated a chain-linked index of railway shares that covered 1834 to 1862 as well as some regional rail indices. A railway index of 25 shares calculated by Frederick R. Macaulay in The Movements of Interest Rates, Bond Yields and Stock Prices in the United States Since 1856, New York: National Bureau of Economic Research, 1938 was used to fill in the gap between 1862 and 1870.
The performance of the rail indices is provided below. Although the GFD-100 and Smith and Cole Rail index take divergent paths, they both end up at the same levels over the 28-year period. The GFD Transports outperform the Smith and Cole Railway Index, especially in the 1850s, but the general pattern of their performance at any point in time is similar. The difference lies in the rate of return.
It is unfortunate that the Smith and Cole indices have been the sole source of returns for the United States before 1870. The problem with the Smith and Cole indices are several: 1) they use a limited number of stocks, 2) their selection process creates a survivorship bias that increases returns, 3) calculations are based upon the geometric mean of the shares rather than the capitalization of each share, 4) railway shares are spliced together to provide consistent coverage, and 5) no data on dividends was collected so no total returns could be calculated.
As was discussed above, because of the survivorship bias that is inherent in the Cole and Smith indices, one would expect that their indices would outperform the other indices that lack a survivorship bias. The survivorship bias is particularly notable in the exclusion of the second Bank of the United States. After the bank lost its charter from the Federal government in 1836, Stephen Girard took over the bank and privatized it. The bank collapsed during the depression that followed the Panic of 1837, and Bank of the United States stock declined in value from 123 in January 1839 to 3.5 by December 1841. The bank’s capitalization fell from $42 million to $1 million. The GFD-100 index lost almost half of its value as a result.
The chart below compares the Smith and Cole/Macaulay index with the GFD-100 index from 1802 until 1871. As can be seen, the survivorship bias enables the Smith and Cole/Macaulay index to outperform the GFD-100 during the entire period. From 1845 on, the Smith and Cole/Macaulay index includes no finance stocks and outperforms the GFD-100 even more. The combination of survivorship bias and exclusion of bank and insurance shares after 1845 causes the GFD-100 to underperform the Smith and Cole/Macaulay index. Because the GFD-100 includes more stocks and is cap-weighted, it provides a better representation of the return shareholders would have received before the introduction of the Cowles indices in 1871.
Limitations of the Cowles Indices
In the 1930s, Cowles created cap-weighted indices for the United States going back to 1871. The Commercial and Financial Chronicle published annual summaries of monthly high and low prices for stocks. The Cowles Commission collected data on shares outstanding and dividends in order to calculate cap-weighted price and total return indices for the United States. The Cowles indices included 44 shares in 1871, 75 in 1880, 88 in 1890, 102 in 1900, 126 in 1910, 243 in 1920 and 450 in 1930. The GFD-100 index included about 100 shares in each of those years.
Although the Cowles indices are cap-weighted and include dividends, there are drawbacks to the Cowles indices. In calculating the indices, prices were based upon the average of the high and low prices for each month, not on the closing price of shares in each month. Although this factor affects the volatility of the indices, it should have a minimal impact on the long-term returns of the index. More importantly, the selection of stocks was limited to shares traded on the New York Stock Exchange. This means that bank and insurance shares as well as Standard Oil and other non-NYSE companies are completely excluded from the Cowles indices. Even though Standard Oil began trading in 1882, neither Standard Oil of New Jersey (ExxonMobil) nor of California (Chevron) was included in the Cowles indices until 1918 when they joined the NYSE. At the time, Standard Oil was the largest corporation in the world.
The impact of excluding Standard Oil, banks and insurance companies is illustrated below.