I broke the process into four steps: acquiring the data, extracting L^aT_eX commands from caption environments, finding potential figure caption candidates, and verifying these candidates. As the arXiv source archive is well over 1 TB in size, it is provided in an AWS S3 bucket configured such that the requester pays for bandwidth, which would result in a bandwidth bill of >$100 if downloaded directly. As I was only interested in the T_eX source and not the figures, which account for most of the total file size, and since AWS does not charge to transfer between S3 buckets and EC2 instances in the same region, I first ran a script on an EC2 instance to download from arXiv’s S3 bucket and extract and repackage just the T_eX source files. This allowed me to greatly reduce the amount of data transfer required and allowed me to download the full T_eX source file corpus for <$5. Next, I used the TexSoup Python package to process the T_eX files and produce a list of L^aT_eX commands used in the caption environment. I then used a final script to search for papers that used command names that referenced colors or shapes to compile a list of likely paper candidates and produced HTML files for each year containing a link to the PDF for each candidate paper as well as the full T_eX source for the identified caption, with the matching commands highlighted. Finally, I manually verified the papers using the HTML files that were produced. Except for trivial false positivies, which could be identified by looking at the included caption source, I manually looked at the PDF for each candidate paper, verified that it included a visual caption indicator, and classified the caption indicator if it had one. For papers that included indicators, I then attempted to locate the published version of record of the paper and did the same for it.

Through this process, my scripts located around ~5100 paper candidates from the beginning of arXiv in 1992 through the end of June 2020. I manually verified these candidates for papers submitted prior to the end of 2016; these accounted for ~2000 candidates, of which I verified ~1100 papers to have some sort of visual caption indicator. For ~700 of these, I was able to verify the presence of some form of visual caption indicator in the published version of record. Of these, ~60% included a black shape or line indicator, ~25% included a color shape or line indicator, and the remainder included colored text. The fraction of papers with color shape or line indicators was higher in the pre-prints, since it was not uncommon for the published version to include a black indicator when the pre-print included a colored indicator. I stopped at the end of 2016 since the verification process was quite time consuming, and I could only look at so many papers before giving up.

These findings show that the idea of using figure color caption indicators is by no means a new idea. However, it’s still quite rare in relative terms, since at most a couple thousand out of arXiv’s ~1.7 million pre-prints include such indicators. Most of the examples I found used a colored shape (■) or line (—) in parentheses, or both in cases where both a line and marker were used. My proposal to use a colored underline does still appear to have been a novel concept, but it proved quite complicated to implement, so using shapes or lines in parentheses is much more practical, since it is simpler and is evidentially compatible with many publishers’ workflows. Furthermore, the existing examples can be used as evidence when complaining about paper proofs, after the typesetter predictably removes the indicators, to show that the indicators are possible and that they can and should be included in the final published version of the paper.

One color indicator that I recommend against using is colored text, since it can be difficult to read and often violates WCAG contrast guidelines. Its use seems particularly common in the computer vision literature and, to a lesser degree, the machine learning literature. It is often used to highlight table entries, a purpose much better served by using italic, bold, or bold–italic text.

I have made the scripts used for this analysis, the paper candidates, and the final verified results available. The final verified results are also available separately for easy viewing. Note that the verified results are incomplete and may contain errors.

I have only made minor changes to the previously detailed analysis procedures (see previous set ranking and order ranking posts for details), but there are now ~50% more responses, which has helped with training stability and has reduced uncertainty between different models in the network ensemble. The figure below shows the fifteen lowest ranked six-color color sets on the left and the fifteen highest ranked six-color color sets on the right.

The accuracy for both the training and tests sets remained at 58%. The plot below shows the average six-color color set scores as a function of rank, with a 1-sigma error band.

Using the highest-ranked six-color set, the figure below shows the fifteen lowest ranked orderings on the left and the fifteen highest ranked orderings on the right.

Accuracy was similar to before, with an accuracy of 55% on the training set and an accuracy of 54% on the test set. The plot below shows the average ordering scores as a function or rank, with a 1-sigma error band.

Next, the same technique was extended to the eight-color color sets. The figure below shows the fifteen lowest ranked eight-color color sets on the left and the fifteen highest ranked eight-color color sets on the right.

The accuracy was 57% for both the training and test sets. The plot below shows the average eight-color color set scores as a function of rank, with a 1-sigma error band.

Using the highest-ranked eight-color set, the figure below shows the fifteen lowest ranked orderings on the left and the fifteen highest ranked orderings on the right.

The accuracy was 55% on the training set and 53% on the test set. The plot below shows the average ordering scores as a function or rank, with a 1-sigma error band.

This is an incremental improvement over the previous results, as it just used extra data, while keeping the analysis procedure the same. The fact that accuracy was similar when the analysis was extended to eight-color color sets and color cycles is promising. I’d like to devise a method that combines both the six-color and eight-color color sets in the training process to maximize the use of the response data; I have a few ideas on how to do this but nothing concrete yet. I’ve also looked more into the idea of devising a color namability criterion by reanalyzing the xkcd Color Survey results. While my reanalysis has led to some interesting tidbits about color names, it didn’t really pan out as far as becoming a useful criterion for ranking the color sets at hand. I’ve been trying to clarify the licensing on the raw xkcd Color Survey responses database dump before writing up my findings, but so far, I have not received a reply from Randall Munroe (which is understandable). As always, more responses would be helpful. I had not originally intended for the survey to go on as long as it has, but as I’ve been busy with my normal (cosmology-related) research and as I’ve not received as many responses as I had hoped for, the survey remains open to responses. I plan on leaving it open until the analysis is close to final (at least a few more months), after which I’ll close the survey to responses and execute the final analysis runs.

Earlier this year, I became aware of a feature in GitHub-flavored Markdown that displays a colored square inline when HTML color codes are surrounded by backticks, e.g., #1f77b4. Although I only recently became aware of this feature, it dates back to at least 2017 and is similar to a feature that Slack has had since at least 2014. When I saw this inline color presentation, I immediately thought of its applicability to figure captions, particularly in academic papers; as a colorblind individual, matching colors referenced in figure captions to features in the figures themselves can be challenging at times due to difficulties with naming colors. Thus, I added similar annotations to figure captions in my recently submitted paper, Two-year Cosmology Large Angular Scale Surveyor (CLASS) Observations: A First Detection of Atmospheric Circular Polarization at Q Band:

Fig. 2. Frequency dependence of polarized atmospheric signal at zenith for the CLASS observing site, both for circular polarization (, shown in blue) and linear polarization (, shown in orange). The light gray bands indicate CLASS observing frequencies, with the lowest frequency band corresponding to the Q-band telescope.

Fig. 5. Example binned azimuth profiles are shown…angle cut. The profile in blue is from a zenith angle of 43.9° and a boresight rotation angle of −45°, the profile in orange is from a zenith angle of 46.7° and a boresight rotation angle of 0°, and the profile in red is from a zenith angle of 52.8° and a boresight rotation angle of +45°.

The first caption refers to a line plot, while the second caption refers to a scatter plot with best fit lines. These examples, as well as underlining examples elsewhere in this post, display best in a browser that supports changing the underline thickness via the text-decoration-thickness CSS property. At the time of writing, this includes Firefox 70+ and Safari 12.2+ but does not include any version of Chrome; however, browser underlining support is still subpar to the underline rendered by , so the reader is encouraged to view the figures in the paper.

While the primary purpose of these annotations is to improve accessibility for individuals with color vision deficiencies, they are also helpful when a paper is printed or displayed in grayscale. For example, it is much easier to distinguish blue and orange in grayscale with the annotations than without.

As this was an experiment, I included two different methods for visualizing the color, a thick colored underline under and a colored square following the color name. Since the colors are referring to solid lines in the plot, the underlines make sense because they match the plot features, e.g., a solid blue line. Likewise, a dotted underline might make sense for a dotted blue line, although it is more difficult to discern the color of the dotted line than the solid line. I am undecided as to whether or not including the colored square is a good idea. While it adds an additional visual cue, the main reason I included it was to increase the chances of at least one of the indicators making it past the editors and into the final published paper; as the paper is currently under review, it remains to be seen if either indicator survives the publication process.

For scatter plots, however, colored shapes make perfect sense. A scatter plot with red squares (), blue diamonds (), and orange circles () should include such shapes in the figure caption when the caption refers to the corresponding points. I am undecided as to whether or not the color names in such cases should be underlined, just as I am undecided as to whether or not line plots should included a colored square. Although I have not seen any color indicators, for either lines or scatter points, in the scientific literature, the use of shapes in figure captions is not a new practice. I have found examples dating from the mid-1950s through the early 2000s. The closest example I have found is in a 1997 paper¹ that refers to a symbol with both its name and a graphical representation:

Fig. 5. Couette-Taylor experiments. Logarithmic…number. The black triangles (▲) are the results obtained with smooth cylinders, and the open ones (△) correspond to those obtained with the ribbed ones. The crosses (×) show for comparison…and Swinney [8].

Other examples include a 1967 paper² (and a 1968 paper³) that uses graphical representations inline instead of symbol names:

Fig. 13. Additional…symmetries. Points marked with ■ are the excess…nuclei, points marked with □ the excess…N = Z. The points ▼ show the differences…larger Z-values. The points △ are the differences…for even-Z–odd-N nuclei.

and a 1955 paper⁴ that puts the figure legend inline in the figure caption:

Fig. 1. (p,n) cross sections in millibarns. ○—measured total…isotope; □—partial…isotope; ×—observed…estimate. Curves…of r₀. The dotted bands indicate…energy.

There are other examples, e.g., this 1960 paper,⁵ that put the legend on separate lines at the end of the caption, but doing so isn’t really the same idea. There are also papers that treated line styles in the same manner as scatter plot symbols, such as this 1962 paper:⁶

Fig. 1. Counting rate…in pulses per cm² sec. Maximum…is indicated by broken lines (– – –). The zone…has been shaded.

These examples should not be considered by any means exhaustive, since searching for this sort of thing is extremely difficult.⁷ In particular, while I don’t know of any prior publications that include color indicators, this does not mean that they do not exist. If anyone reading this is aware of any such examples, or of other interesting figure caption indicators, please let me know.

Adoption of visual color indicators such as the ones presented here would be a significant accessibility improvement, but it would require buy-in from both publishers and authors. The chances of success are unclear but would certainly be improved with advocacy.

Implementation

The color annotation command was defined as

% Black square
\usepackage{amsmath}

% Define color
\usepackage{xcolor}
\definecolor{tab:blue}{RGB}{31, 119, 180}

% Color underlines with breaks for descenders, based on:
% https://tex.stackexchange.com/a/75406
% https://tex.stackexchange.com/a/24771
% https://tex.stackexchange.com/a/321235
\usepackage{soul}
\usepackage[outline]{contour}
\newcommand \colorindicator[2]{%
  \begingroup%
  \setul{0.25ex}{0.4ex}%
  \contourlength{0.2ex}%
  \setulcolor{#1}%
  \ul{{\phantom{#2}}}\llap{\contour{white}{#2}} \textcolor{#1}{\tiny{$\blacksquare$}}%
  \endgroup%
}

and used with \colorindicator{tab:blue}{blue}. For HTML, this CSS

.color-underline {
  text-decoration-line: underline;
  text-decoration-style: solid;
  text-decoration-thickness: 0.2em;
  text-decoration-skip-ink: auto;
}
.blue {
  text-decoration-color: #1f77b4;
}
.blue-square::after {
  content: "\202f\25a0";
  position: relative;
  display: inline-block;
  color: #1f77b4;
}

was used with <span class="color-underline blue blue-square">blue</span> to produce blue. A production implementation would probably involve a symbol web font to improve and normalize the symbol appearance and possibly a better way to draw underlines.

Update (2020-10-31): see update on search for existing examples

1. Cadot, O., Y. Couder, A. Daerr, S. Douady, and A. Tsinober. “Energy injection in closed turbulent flows: Stirring through boundary layers versus inertial stirring.” Physical Review E 56, no. 1 (1997): 427. doi:10.1103/PhysRevE.56.427  ↩

2. Haque, Khorshed Banu, and J. G. Valatin. “An investigation of the separation energies of lighter nuclei.” Nuclear Physics A 95, no. 1 (1967): 97-114. doi:10.1016/0375-9474(67)90154-6  ↩

3. Aydin, C. “The spectral variations of CU Virginis (HD 124224).” Memorie della Societa Astronomica Italiana 39 (1968): 721. bibcode:1968MmSAI..39..721A  ↩

4. Blosser, H. G., and T. H. Handley. “Survey of (p, n) reactions at 12 MeV.” Physical Review 100, no. 5 (1955): 1340. doi:10.1103/PhysRev.100.1340  ↩

5. Evans, D. S., G. V. Raynor, and R. T. Weiner. “The lattice spacings of thorium-lanthanum alloys.” Journal of Nuclear Materials 2, no. 2 (1960): 121-128. doi:10.1016/0022-3115(60)90039-8  ↩

6. Vernov, S. N., E. V. Gorchakov, Yu I. Logachev, V. E. Nesterov, N. F. Pisarenko, I. A. Savenko, A. E. Chudakov, and P. I. Shavrin. “Investigations of radiation during flights of satellites, space vehicles and rockets.” Journal of the Physical Society of Japan Supplement 17 (1962): 162. bibcode:1962JPSJS..17B.162V  ↩

7. I found most of the above examples by performing full-text searches in NASA ADS for terms such as “black diamond” or “filled square” and looking through hundreds of results to find the few instances that included both the search terms and the symbols.  ↩

Since rainbow colormaps are best suited for quickly judging values, their most important property is that colors in non-adjacent sections of the colormap are not confused.⁴ To evaluate this quantitatively, I devised the following metric. For each color in the colormap, the perceptual distance in CAM02-UCS⁵ is calculated for every additional color in the colormap. The weighted average of the perceptual distances is then taken, with the squares of the color location distances in the colormap used as weights. For color vision deficiencies, the method of Machado et al. (2009) is used to adjust the colors before the perceptual distance is calculated, as I did for randomly generating color sets and as was done in the development of Cividis;⁶ a severity of 100 was used, indicating deuteranopia, protanopia, and tritanopia. Thus, similar colors in distant locations in the colormap are penalized.

We will start with rainbow colormaps for our evaluation of colormaps by this metric, first considering Jet, the new Turbo colormap, and Matplotlib’s existing Rainbow colormap, which also attempts to address some of Jet’s shortcomings. In the plot legends, the abbreviations “Norm,” “Deut,” “Prot,” and “Trit” are used for normal color vision, deuteranopia, protanopia, and tritanopia, respectively. Higher perceptual distance, ΔE, is better, as are smoother and more consistent discernibility lines.

The discernibility lines for Turbo and Rainbow are much smoother than those for Jet, since both mitigate Jet’s significant banding issues. Although Jet’s banding issues are generally considered problematic, I, as a colorblind individual, find the banding to sometimes be a redeeming quality, since it makes it easier for me to match part of an image to the colorbar or other parts of the image. For normal color vision, Turbo’s discernibility line is smooth and fairly flat, a significant improvement over Jet, and a minor improvement over Rainbow, although Turbo arguably looks better. However, the discernibility lines for various color vision deficiencies are not nearly as uniform, for either Turbo or Rainbow. This means that for colorblind individuals some parts of the colormaps are considerably more difficult to discern than others, making data plotted with them liable to misinterpretation. Thus, while Turbo and Rainbow improve upon some of Jet’s shortcomings, neither is colorblind-friendly.

Next, we will consider cyclic rainbow colormaps. The classic, and severely flawed, version is the HSV colormap, and the improved version is Sinebow; in regards to non-cyclic rainbow colormaps, these are analogous to Jet and Turbo, respectively. Twilight, a perceptually uniform cyclic colormap, is also considered.

In evaluating the metric for these colormaps, their cyclic nature was taken into consideration in the colormap location distance calculation. Sinebow’s discernibility lines are much smoother than HSV’s, but neither does well for color vision deficiencies. Twilight is much more consistent and colorblind-friendly, although at the expense of average discernibility.

Now, we will consider two perceptually uniform linear colormaps, Viridis, the Matplotlib default, and Cividis a derivative designed with color vision deficiencies in mind.

The “V” shape of the metric for these colormaps is expected, since for a linear colormap, the center is closest to the greatest number of other colors. Note that the discernibility of Cividis, which was optimized with color vision deficiencies in mind, is the most consistent between normal color vision and various color vision deficiencies, although Viridis is also okay in this regard, and both are considerably better than any of the rainbow colormaps previously presented.

Finally, diverging colormaps will be evaluated. Here, we consider Matplotlib’s Coolwarm colormap and Peter Kovesi’s Blue-Gray-Yellow colormap.

These show a “V” shape, similar to linear colormaps, although this is less pronounced in Coolwarm. The Blue-Gray-Yellow colormap is linearly increasing in lightness and perceptually uniform, so its discernibility profile is much closer to that of perceptually uniform linear colormaps.

In summary, while Turbo does ameliorate many of the issues with Jet, neither Turbo nor any of the other rainbow colormaps evaluated here are colorblind-friendly, at least per the metric evaluated. It is likely that it is not possible to construct a rainbow colormap with such a property, unlike for linear, diverging, and cyclic colormaps. The Jupyter notebook used to evaluate the colormaps and produce the plots is available.

Edit (2020-07-15): Replaced last plot (and updated Jupyter notebook), since the non-diverging BGY colormap had been accidentally used originally instead of the BJY colormap. All plots were also updated for improved readability.

1. It probably is possible to create a colorblind-friendly rainbow colormap for a particular type of color vision deficiency. However, creating such a colormap that simultaneously works for multiple types of color vision deficiencies as well as for normal color vision is what is likely impossible.  ↩

2. G. W. Meyer and D. P. Greenberg, “Color-defective vision and computer graphics displays,” in IEEE Computer Graphics and Applications, vol. 8, no. 5, pp. 28-40, Sept. 1988. doi:10.1109/38.7759  ↩

3. G. M. Machado, M. M. Oliveira, and L. A. F. Fernandes, “A Physiologically-based Model for Simulation of Color Vision Deficiency,” in IEEE Transactions on Visualization and Computer Graphics, vol. 15, no. 6, pp. 1291-1298, Nov.-Dec. 2009. doi:10.1109/TVCG.2009.113  ↩

4. When differences between adjacent colors are important, a perceptually uniform colormap should be used.  ↩

5. Luo M.R., Li C. (2013) CIECAM02 and Its Recent Developments. In: Fernandez-Maloigne C. (eds) Advanced Color Image Processing and Analysis. Springer, New York, NY. doi:10.1007/978-1-4419-6190-7_2  ↩

6. J. R. Nuñez, C. R. Anderton, and R. S. Renslow. “Optimizing colormaps with consideration for color vision deficiency to enable accurate interpretation of scientific data,” in PLoS ONE vol. 13, no. 7, pp. e0199239, Aug. 2018. doi:10.1371/journal.pone.0199239  ↩

To maximize the information gleaned from the survey responses, the network was trained in four steps. The process started with a single network and ended with a conjoined network, as before, except the single network underwent three stages of training instead of one. First, the color set responses—the responses that were used in the previous analysis—were used to train the network for 50 epochs, to learn color representations. Next, the ordering responses were used with the data augmented with all possible cyclic shifts to train the network for an additional 50 epochs, to learn internal cycle orderings. Then, the non-augmented ordering responses were used to train the network for another 100 epochs, to learn the ideal starting color. Finally, the last layer of the network was replaced, as before, to make a conjoined network, and the new network was trained for a final 100 epochs, again with the non-augmented ordering responses.

As with the previous analysis, an ensemble of 100 network instantiations was trained, and the average and standard deviation of the scores were computed. The accuracy for the ordering was a bit worse than for the color sets, with an accuracy of 56% on the training data and an accuracy of 54% on the test data. Since the ideal ordering depends on the specific color set used, the highest ranked color set from the previous analysis was used in this evaluation. The error band from the trained ensemble for this color set was larger than the error band from the set ranking analysis. While the model could be evaluated for any color set, it is likely more accurate for color sets that were ranked highly in the previous analysis, since the Color Cycle Survey only asks the user about the preferred ordering of the user’s preferred color set, so data are not collected on poorly-liked color sets.

The trained network shows a clear preference for blue / purple as the first color instead of green / yellow; as many existing color cycles start with blue, this seems reasonable. The network also seems fairly confident in picking the third color, since it’s the same for the top fifteen orderings, but there’s more variation in the second color.