I broke the process into four steps: acquiring the data, extracting L^aT_eX commands from caption environments, finding potential figure caption candidates, and verifying these candidates. As the arXiv source archive is well over 1 TB in size, it is provided in an AWS S3 bucket configured such that the requester pays for bandwidth, which would result in a bandwidth bill of >$100 if downloaded directly. As I was only interested in the T_eX source and not the figures, which account for most of the total file size, and since AWS does not charge to transfer between S3 buckets and EC2 instances in the same region, I first ran a script on an EC2 instance to download from arXiv’s S3 bucket and extract and repackage just the T_eX source files. This allowed me to greatly reduce the amount of data transfer required and allowed me to download the full T_eX source file corpus for <$5. Next, I used the TexSoup Python package to process the T_eX files and produce a list of L^aT_eX commands used in the caption environment. I then used a final script to search for papers that used command names that referenced colors or shapes to compile a list of likely paper candidates and produced HTML files for each year containing a link to the PDF for each candidate paper as well as the full T_eX source for the identified caption, with the matching commands highlighted. Finally, I manually verified the papers using the HTML files that were produced. Except for trivial false positivies, which could be identified by looking at the included caption source, I manually looked at the PDF for each candidate paper, verified that it included a visual caption indicator, and classified the caption indicator if it had one. For papers that included indicators, I then attempted to locate the published version of record of the paper and did the same for it.

Through this process, my scripts located around ~5100 paper candidates from the beginning of arXiv in 1992 through the end of June 2020. I manually verified these candidates for papers submitted prior to the end of 2016; these accounted for ~2000 candidates, of which I verified ~1100 papers to have some sort of visual caption indicator. For ~700 of these, I was able to verify the presence of some form of visual caption indicator in the published version of record. Of these, ~60% included a black shape or line indicator, ~25% included a color shape or line indicator, and the remainder included colored text. The fraction of papers with color shape or line indicators was higher in the pre-prints, since it was not uncommon for the published version to include a black indicator when the pre-print included a colored indicator. I stopped at the end of 2016 since the verification process was quite time consuming, and I could only look at so many papers before giving up.

These findings show that the idea of using figure color caption indicators is by no means a new idea. However, it’s still quite rare in relative terms, since at most a couple thousand out of arXiv’s ~1.7 million pre-prints include such indicators. Most of the examples I found used a colored shape (■) or line (—) in parentheses, or both in cases where both a line and marker were used. My proposal to use a colored underline does still appear to have been a novel concept, but it proved quite complicated to implement, so using shapes or lines in parentheses is much more practical, since it is simpler and is evidentially compatible with many publishers’ workflows. Furthermore, the existing examples can be used as evidence when complaining about paper proofs, after the typesetter predictably removes the indicators, to show that the indicators are possible and that they can and should be included in the final published version of the paper.

One color indicator that I recommend against using is colored text, since it can be difficult to read and often violates WCAG contrast guidelines. Its use seems particularly common in the computer vision literature and, to a lesser degree, the machine learning literature. It is often used to highlight table entries, a purpose much better served by using italic, bold, or bold–italic text.

I have made the scripts used for this analysis, the paper candidates, and the final verified results available. The final verified results are also available separately for easy viewing. Note that the verified results are incomplete and may contain errors.

Earlier this year, I became aware of a feature in GitHub-flavored Markdown that displays a colored square inline when HTML color codes are surrounded by backticks, e.g., #1f77b4. Although I only recently became aware of this feature, it dates back to at least 2017 and is similar to a feature that Slack has had since at least 2014. When I saw this inline color presentation, I immediately thought of its applicability to figure captions, particularly in academic papers; as a colorblind individual, matching colors referenced in figure captions to features in the figures themselves can be challenging at times due to difficulties with naming colors. Thus, I added similar annotations to figure captions in my recently submitted paper, Two-year Cosmology Large Angular Scale Surveyor (CLASS) Observations: A First Detection of Atmospheric Circular Polarization at Q Band:

Fig. 2. Frequency dependence of polarized atmospheric signal at zenith for the CLASS observing site, both for circular polarization (, shown in blue) and linear polarization (, shown in orange). The light gray bands indicate CLASS observing frequencies, with the lowest frequency band corresponding to the Q-band telescope.

Fig. 5. Example binned azimuth profiles are shown…angle cut. The profile in blue is from a zenith angle of 43.9° and a boresight rotation angle of −45°, the profile in orange is from a zenith angle of 46.7° and a boresight rotation angle of 0°, and the profile in red is from a zenith angle of 52.8° and a boresight rotation angle of +45°.

The first caption refers to a line plot, while the second caption refers to a scatter plot with best fit lines. These examples, as well as underlining examples elsewhere in this post, display best in a browser that supports changing the underline thickness via the text-decoration-thickness CSS property. At the time of writing, this includes Firefox 70+ and Safari 12.2+ but does not include any version of Chrome; however, browser underlining support is still subpar to the underline rendered by , so the reader is encouraged to view the figures in the paper.

While the primary purpose of these annotations is to improve accessibility for individuals with color vision deficiencies, they are also helpful when a paper is printed or displayed in grayscale. For example, it is much easier to distinguish blue and orange in grayscale with the annotations than without.

As this was an experiment, I included two different methods for visualizing the color, a thick colored underline under and a colored square following the color name. Since the colors are referring to solid lines in the plot, the underlines make sense because they match the plot features, e.g., a solid blue line. Likewise, a dotted underline might make sense for a dotted blue line, although it is more difficult to discern the color of the dotted line than the solid line. I am undecided as to whether or not including the colored square is a good idea. While it adds an additional visual cue, the main reason I included it was to increase the chances of at least one of the indicators making it past the editors and into the final published paper; as the paper is currently under review, it remains to be seen if either indicator survives the publication process.

For scatter plots, however, colored shapes make perfect sense. A scatter plot with red squares (), blue diamonds (), and orange circles () should include such shapes in the figure caption when the caption refers to the corresponding points. I am undecided as to whether or not the color names in such cases should be underlined, just as I am undecided as to whether or not line plots should included a colored square. Although I have not seen any color indicators, for either lines or scatter points, in the scientific literature, the use of shapes in figure captions is not a new practice. I have found examples dating from the mid-1950s through the early 2000s. The closest example I have found is in a 1997 paper¹ that refers to a symbol with both its name and a graphical representation:

Fig. 5. Couette-Taylor experiments. Logarithmic…number. The black triangles (▲) are the results obtained with smooth cylinders, and the open ones (△) correspond to those obtained with the ribbed ones. The crosses (×) show for comparison…and Swinney [8].

Other examples include a 1967 paper² (and a 1968 paper³) that uses graphical representations inline instead of symbol names:

Fig. 13. Additional…symmetries. Points marked with ■ are the excess…nuclei, points marked with □ the excess…N = Z. The points ▼ show the differences…larger Z-values. The points △ are the differences…for even-Z–odd-N nuclei.

and a 1955 paper⁴ that puts the figure legend inline in the figure caption:

Fig. 1. (p,n) cross sections in millibarns. ○—measured total…isotope; □—partial…isotope; ×—observed…estimate. Curves…of r₀. The dotted bands indicate…energy.

There are other examples, e.g., this 1960 paper,⁵ that put the legend on separate lines at the end of the caption, but doing so isn’t really the same idea. There are also papers that treated line styles in the same manner as scatter plot symbols, such as this 1962 paper:⁶

Fig. 1. Counting rate…in pulses per cm² sec. Maximum…is indicated by broken lines (– – –). The zone…has been shaded.

These examples should not be considered by any means exhaustive, since searching for this sort of thing is extremely difficult.⁷ In particular, while I don’t know of any prior publications that include color indicators, this does not mean that they do not exist. If anyone reading this is aware of any such examples, or of other interesting figure caption indicators, please let me know.

Adoption of visual color indicators such as the ones presented here would be a significant accessibility improvement, but it would require buy-in from both publishers and authors. The chances of success are unclear but would certainly be improved with advocacy.

Implementation

The color annotation command was defined as

% Black square
\usepackage{amsmath}

% Define color
\usepackage{xcolor}
\definecolor{tab:blue}{RGB}{31, 119, 180}

% Color underlines with breaks for descenders, based on:
% https://tex.stackexchange.com/a/75406
% https://tex.stackexchange.com/a/24771
% https://tex.stackexchange.com/a/321235
\usepackage{soul}
\usepackage[outline]{contour}
\newcommand \colorindicator[2]{%
  \begingroup%
  \setul{0.25ex}{0.4ex}%
  \contourlength{0.2ex}%
  \setulcolor{#1}%
  \ul{{\phantom{#2}}}\llap{\contour{white}{#2}} \textcolor{#1}{\tiny{$\blacksquare$}}%
  \endgroup%
}

and used with \colorindicator{tab:blue}{blue}. For HTML, this CSS

.color-underline {
  text-decoration-line: underline;
  text-decoration-style: solid;
  text-decoration-thickness: 0.2em;
  text-decoration-skip-ink: auto;
}
.blue {
  text-decoration-color: #1f77b4;
}
.blue-square::after {
  content: "\202f\25a0";
  position: relative;
  display: inline-block;
  color: #1f77b4;
}

was used with <span class="color-underline blue blue-square">blue</span> to produce blue. A production implementation would probably involve a symbol web font to improve and normalize the symbol appearance and possibly a better way to draw underlines.

Update (2020-10-31): see update on search for existing examples

1. Cadot, O., Y. Couder, A. Daerr, S. Douady, and A. Tsinober. “Energy injection in closed turbulent flows: Stirring through boundary layers versus inertial stirring.” Physical Review E 56, no. 1 (1997): 427. doi:10.1103/PhysRevE.56.427  ↩

2. Haque, Khorshed Banu, and J. G. Valatin. “An investigation of the separation energies of lighter nuclei.” Nuclear Physics A 95, no. 1 (1967): 97-114. doi:10.1016/0375-9474(67)90154-6  ↩

3. Aydin, C. “The spectral variations of CU Virginis (HD 124224).” Memorie della Societa Astronomica Italiana 39 (1968): 721. bibcode:1968MmSAI..39..721A  ↩

4. Blosser, H. G., and T. H. Handley. “Survey of (p, n) reactions at 12 MeV.” Physical Review 100, no. 5 (1955): 1340. doi:10.1103/PhysRev.100.1340  ↩

5. Evans, D. S., G. V. Raynor, and R. T. Weiner. “The lattice spacings of thorium-lanthanum alloys.” Journal of Nuclear Materials 2, no. 2 (1960): 121-128. doi:10.1016/0022-3115(60)90039-8  ↩

6. Vernov, S. N., E. V. Gorchakov, Yu I. Logachev, V. E. Nesterov, N. F. Pisarenko, I. A. Savenko, A. E. Chudakov, and P. I. Shavrin. “Investigations of radiation during flights of satellites, space vehicles and rockets.” Journal of the Physical Society of Japan Supplement 17 (1962): 162. bibcode:1962JPSJS..17B.162V  ↩

7. I found most of the above examples by performing full-text searches in NASA ADS for terms such as “black diamond” or “filled square” and looking through hundreds of results to find the few instances that included both the search terms and the symbols.  ↩