Stanford Workshop on Medical VR and AR

5 April 2018, there will be a public workshop on medical head-mounted displays in Stanford. The workshop is designed to support collaborations between the engineers who are developing VR and AR technologies and the surgeons and clinicians who are using these technologies to treat their patients.

The workshop features talks by researchers who are developing VR and AR technologies to advance healthcare and panel discussions with Stanford physicians who are using VR and AR applications for surgical planning and navigation and for alleviating pain and anxiety in their patients.

There will be an interactive demo session featuring research projects, clinical applications, and startup ventures.

Seating is limited, so if you wish to attend, we recommend that you register now at the website https://scien.stanford.edu/index.php/medicalvrar.

Retinotopy

The latest issue of Science magazine has an article explaining how the retinotopic map is built during the development of the eye. The authors show that glial cells that ensheath axons relay cues from photoreceptors to induce the differentiation of the photoreceptor target field—the so-called lamina neurons. Thus, glia can play an instructive role in differentiation, helping to direct the spatiotemporal patterning of neurogenesis.

Science 01 Sep 2017: Vol. 357, Issue 6354, pp. 886–891, DOI: 10.1126/science.aan3174

Another recent article demonstrated that there is no retinotopic map further up in the visual system where object recognition takes place.

Science 18 Aug 2017: Vol. 357, Issue 6352, pp. 687-692, DOI: 10.1126/science.aan4800

biology of color

The 4 August issue of Science (Vol. 357, Issue 6350, eaan0221) has a valuable article on the biology of color describing the current state of the art in this interdisciplinary field of animal coloration. This article is important because in the past 20 years there has been significant progress in this field.

Regenerating optic pathways

Less than a mile down Embarcadero Road from Newell Road is Stanford University's Ophthalmology department. In Science Vol. 356, Issue 6342, pp. 1031–1034, three researchers from the School of Medicine report on the current status in retinal ganglion cell (RGC, pink in the figure) regeneration. When the optic nerve is severed, for example after an accident or with glaucoma, the retinal ganglion cells quickly die off. Even if the rest of the retina remains intact, sight is lost.

The retinal ganglion cells are part of the central nervous system, thus unlike in the peripheral nervous system, severed axons do not regenerate. After injury and inflammation, in the eye, there is a balance of activating and inhibiting factors. For example, amacrine cells (orange in the figure) release zinc, which is an inhibitor, while the lens can cause macrophages to release oncomodulin, a protein that promotes RGC axon extension. The challenge is to understand these balancing mechanisms. A further challenge is to regrow the axons correctly all the way to the lateral geniculate nucleus (LGN).

The authors outline three possible avenues for restoring the RGCs and thus sight.

To bees, edges are green

In color science, we like to start from spectral data. To obtain the relative response in a photoreceptor, we multiply the reflectance function of a stimulus with the illuminant spectrum and then integrate over the visual spectral range using the receptor's spectral sensitivity function as the integration measure, up to a normalization factor.

Most often, what changes are the stimuli. Sometimes, we change the illuminant to predict the response under a different light source. When we study the response of people with color vision deficiencies, we swap the spectral sensitivity functions, for example, we shift the peak frequencies of the M or L catch probabilities to simulate deuteranomaly respectively protanomaly. For people with normal color vision, the standard values for the peak sensitivities are approximately 430 nm (S-cones), 540 nm (M-cones), and 570 nm (L-cones).

The approach is not limited to humans. For example, bees also have three receptors, with peak sensitivities at 344 nm (S), 436 nm (M), and 544 nm (L): their visual spectrum is shifted towards the ultraviolet. If we taught color naming to bees, their red would correspond to our green. Actually, looking at the honeybee (Apis mellifera) sensitivity functions, their color vision is different from ours because they have a secondary peak in the UV region. With only 10,000 ommatidia, their vision also has a much lower spatial resolution.

In their paper Multispectral images of flowers reveal the adaptive significance of using long-wavelength-sensitive receptors for edge detection in bees, Vera Vasas et al. use a collection of multispectral photographs of flowers preferred by bees.

Assuming bees and flowers coevolved to maximize pollination, the authors perform an interesting statistical analysis of what bees would see, to determine under what condition they can best recognize the flower's center areas where the nectar and the stamens/carpels are located. An important boundary condition is that the process has to work in the presence of movement, as flowers are swayed by Zephyr.

The statistical analysis suggests that bees use only the L-receptors to identify edges and segment the visual field and detect movement. This process is different from us humans who use the M- and L-receptors to analyze an essentially monochromatic image.

Citation: Vasas, V., Hanley, D., Kevan, P.G. et al. J Comp Physiol A (2017) 203: 301. doi:10.1007/s00359-017-1156-x

Breaking the barriers to true augmented reality

Today, when you run a job on a digital press, you just turn it on, load the stock, and start printing. An army of sensors and feedback loops work with mathematical models will set up the press. In the heydays of color printing, the situation was very different: skilled press operators would spend hours making ready the press, with only the use of a densitometer and their eyes. It took them years of experience to achieve their master status.

A big breakthrough came when in 1968 when Felix Brunner invented the print control strip, which made press make-ready more of a technical process instead of a magic ceremony. Felix Brunner lived in Corippo, Val Verzasca.

Corippo is a beautiful village but it had been abandoned—people like Michael Silacci of the Opus One Winery, whose grandparents had come to California and never went back. Corippo is still the smallest municipality in Switzerland, with a population of just 13.

Corippo is so stunning, in 1975 it became a protected heritage village. This was quite difficult because the village had become dilapidated. Switzerland raised the funds to transform it into a state-of-the-art modern village that would attract a sophisticated population like Felix Brunner. The challenge was to rebuild it to modern architectural standards without changing its atmosphere and look.

The architecture department at the ETH in Zurich build a 3D model of the entire village, then one by one they started rebuilding the interiors of the houses to the state-of-the-art. The department acquired an Evans and Sutherland Picture System and at each planning step, the commission walked through the virtual village to ascertain that nothing changed the spirit outdoors. For example, if a roof was raised, it was not allowed to cast new and unexpected shadows. If a window was changed, the character of the street was not allowed to change for a passerby, and the view had to feel original from any window.

Although the Picture System was limited to 35,000 polygons, the experience was truly impressive for the planners. If you have a chance to visit Corippo, you will be surprised by the realization. The system was such a breakthrough for urbanists, that Unesco used it for the restoration of Venice. I was also sufficiently impressed to sit down and implement an interactive 3D rendering system, although on the PDP-11with 56 KB of memory running RT-11, I could only display wireframes.

My next related experience was in 1993 when Canon had developed a wearable display and was looking for an acquirer of the rendering software. While the 1975 system for Corippo was rendering coarse polygons, by early 1990 it was possible to do ray tracing, although using an SGI RealityEngine for each eye. An application was to train astronauts for building a space station.

On the quest of finding an interested party for the software, I had the chance to visit almost all companies in the San Francisco Bay Area who were developing wearable displays. On one side, using ray tracing instead of rendering plain solid color polygons made the scene feel more natural, but the big advantage over the Picture System was to be immersed in the virtual scene instead of looking at a display.

There were still quite a few drawbacks. For one, the helmets felt like they were made of lead. The models were still crude because to follow the head movements, ideally, the refresh rate should have been 90 Hz, but even with simple scenes, the refresh rate was typically just 15 or 30 Hz. However, the worst perceptual problem was the lag, which disabled the physiological equilibrium system and caused motion sickness. Another positive development was the transition from the dials and joysticks of 1975 to gloves providing a haptic user interface.

People from my generation spent 13 years in school learning technical drawing, which allows us to visualize mentally a 3D scene from three orthographic projections or from an axonometric projection with perspective. However, in general, understanding a 3D scene from projections is difficult for most people. The value of an immersive display is that you can move your head and thus more easily decode the scene. Consequently, there is still a high interest in wearable displays.

Today, a decent smartphone with CPU, GPU, and DSP has sufficient computing power to do all the rendering necessary for a wearable display. The electronic is so light that it can be fit in a pair of big spectacles that are relatively comfortable to wear and are affordable for professionals to buy. Last year, Bernard Kress had predicted that 2017 would be the year of the wearable display, with dozens of brands and prices affordable by consumers. Why is it not happening?

On March 14, 2017, Prof. Christian Sandor of the Nara Institute of Science and Technology (NAIST) gave a talk with title Breaking the Barriers to True Augmented Reality at SCIEN in Stanford, where he suggested the problem might be that today's developers are not able to augment reality so that the viewer cannot tell what is real. He showed the example of Burnar, where flames are mixed with the user's hands and these users had to interrupt the experiment because their hands were feeling too hot.

True AR has the following two requirements:

1. undetectable modification of user's perception
2. goal: seamless blend of real and virtual world

On a line from manipulating atoms with controlled matter to manipulating perception with implanted AR, current systems should achieve surround AR (full light field display) or personalized AR (perceivable subset). In a full light-field display, the display functions as a window, but with the problem of matching accommodation and vergence. Personalized AR is a smarter approach because the human visual system is measured and only a subset of the light-field is generated, reducing the required display pixels by several orders of magnitude.

In many current systems, the part of the image generated from a computer model is just rendered as a semitransparent blue rendering, hence it is perceived as separate from the real world. True AR requires a seamless blend. The most difficult step is the alignment calibration with the single point active alignment method (SPAAM). The breakthrough from NAIST is that they need to perform SPAAM only once: after that, they use eye tracking for calibration.

The technology is hard to implement. The HoloLens has solved the latency problem, but Microsoft has invested thousands of man-years in developing the system. The optics are very difficult and there are only a few universities teaching it.

yellow may tire autistic children

A research team including Nobuo Masataka, a professor at Kyoto University’s Primate Research Institute, has found that boys with autism spectrum disorder (ASD) tend not to like yellow but show a preference for green. “Yellow may tire autistic children. I want people to take this into account when they use the color on signboards and elsewhere,” Masataka said.

The team, also including France’s University of Rennes 1, has confirmed the color preference of boys with the disorder, according to an article recently published in the journal Frontiers in Psychology. In the study, the color preference of 29 autistic boys aged 4 to 17 was compared with that of 38 age-matched typically developing (TD) boys. All participants were recruited in France, which has clear diagnostic criteria for autism spectrum disorder.

Shown cards of six colors—red, blue, yellow, green, brown and pink—the children were asked to answer which color they like. Yellow was liked by TD boys without the disorder but far less preferred by ASD boys. On the other hand, green and brown were liked more by boys in the ASD group than by those in the TD group, while red and blue were favored to similar degrees by both groups of boys. Pink was unpopular in both groups.

Given the relatively small sample size in each of the three age groups, the failure to find any difference in preference scores between TD children and children with ASD with regard to red, blue and pink might be attributable to a ceiling/floor effect.

The article said yellow has the highest luminance value among the six colors. “The observed aversion to this color might reflect hypersensitivity” of children with ASD, the article said. There is also a general consensus that yellow is the most fatiguing color. When yellow is perceived, both L and M must be involved. The perception of yellow should thus be the most heavily sensory-loaded of the perception of any type of color. Its perception is bearable for TD children but could be over-loaded for children with ASD whose sensitivity to sensory stimulation is enhanced.

Marine Grandgeorge and Nobuo Masataka: "Atypical Color Preference in Children with Autism Spectrum Disorder," Front. Psychol., 23 December 2016, https://doi.org/10.3389/fpsyg.2016.01976

Designing and assessing near-eye displays to increase user inclusivity

Today Emily Cooper, Psychological and Brain Sciences Department at Dartmouth College, gave a talk on designing and assessing near-eye displays to increase user inclusivity. A near-eye display is a wearable display, for example, an augmented reality (AR) or a virtual reality (VR) display.

With most near-eye displays it is not possible or recommended to wear glasses. Some displays, like the HTV Vive, have available lenses to correct the accommodation. We do want to integrate flexible correction into near-eye displays. This can be achieved with a liquid polymer lens with a membrane that can be tuned.

In her lab, for the refraction self-test, the presenter uses an EyeNetra auto-refractometer, which is controlled with a smartphone.

The near-eye display correction is as good as with contact lenses, both in sharpness and in fusion correction. Therefore, it is not necessary to make users wear their correction glasses.

There are two factors determining the image quality of a near-eye display: accommodation and vergence. The problems with incorrect vergence are that users get tired after 20 minutes and the reaction time is slower when the vergence is incorrect.

The solution is to use tunable optics to match the user's visual shortcomings.

A different problem is presbyopia, which is a range reduction. For people older than 45 years, an uncorrected stereo display provides better image quality than correcting the accommodation. However, tunable optics provide better vergence for older people.

A harder problem are people with low vision, regardless of their age. In her lab, Emily Cooper investigated whether consumer-grade augmented reality displays are good enough to help users with low vision.

She used the HoloLens, in which the depth camera in the NIR domain is the key feature to address this problem. Her proposal is to overlay the depth information as a luminance map over the image so that near objects are light and far objects are dark. This allows the users to get by with their residual vision.

Instead of a luminance overlay, a color overlay also works. In this approach, the hue is changed on a segment from warm to cold colors in dependence of their distance. She also tried to encode depth with flicker but is does not work well.

With the HoloLens, it is possible to integrate OCR in the near-eye display and then read all text in the field of view using the 4 speakers in the HoloLens, making the sound come from the location where the text is written.

Progress in wearable displays

Yesterday afternoon, Bernard Kress, Partner Optical Architect at Microsoft Corp, in the HoloLens project, gave a talk at the Stanford Center for Image Systems Engineering (SCIEN) with the title "Human-centric optical design: a key for next generation AR and VR optics." Here is the abstract:

The ultimate wearable display is an information device that people can use all day. It should be as forgettable as a pair of glasses or a watch, but more useful than a smartphone. It should be small, light, low-power, high-resolution and have a large field of view (FOV). Oh, and one more thing, it should be able to switch from VR to AR.

These requirements pose challenges for hardware and, most importantly, optical design. In this talk, I will review existing AR and VR optical architectures and explain why it is difficult to create a small, light and high-resolution display that has a wide FOV. Because comfort is king, new optical designs for the next-generation AR and VR system should be guided by an understanding of the capabilities and limitations of the human visual system.

There are three kinds of wearable displays:

• Smart eyewear: extension of eyewear. Example: Google Glass
• Augmented reality (AR) and mixed reality (MR): extension of the computer. An MR display has a built-in 3d scanner to create a 3d model of the world
• Virtual reality (VR): extension of the gaming console

Bernard surveyed all avenues in wearable displays from their inception to the projections in the future. The speed of the presentation and the amount of material made it impossible to follow the talk unless you are an expert in the field. After the presentation, Bernard told me the size of his PowerPoint file is about 250 MB!

My takeaway was that the biggest issue in wearable displays is cost. So far, the optics engineers designed with cameras in mind and over-designed. The current breakthrough is that now the optics engineers start understanding the HVS, so they can design systems that are just as good as our MTF. Bernard claims that so far the industry has been mostly about hype but in 2017, products will take off and the new challenge is "show me the money."

Extreme color naming experiment finds locus for luma–chroma transformation

From time to time, a new physiological experimentation technique or a significant new instrument is developed, leading to breakthrough discoveries. In the case of color vision, this usually entails a doctoral student and their significant other—and maybe some additional dedicated colleagues or the professor—to undergo the cruel ordeal of having their pupils dilated (mydriasis) and their ciliary muscle paralyzed to avoid accommodation (cycloplegia), then get strapped to a headrest while biting a dental impression mount, to make observations in repeated interminable sessions for months on end, all in the name of science.

In their recent paper The elementary representation of spatial and color vision in the human retina, Ramkumar Sabesan et al. report on a seminal study to locate where in the human visual system (HVS), the luma-chroma encoding occurs in the parvocellular pathway (midget ganglion cells).

This study by Ramkumar Sabesan et al. represents the first time cone photoreceptors of known spectral type have been individually targeted and activated with light in the living human retina.

I cannot believe, it has been thirty years since I drew this diagram:

Although the above diagram looks like a model for the HVS, it was more a plan for my implementation of a color workbench. To keep the head cool and prevent it from overheating, our brain evolved to minimize the usage of energy. This is accomplished by having pipelines where at each stage the information gets recoded to make it more complex but more compact. This is at the cost of speed: while a two-photon catch the shift in electron density takes less than a femtosecond, the entire photo-cycle lasts a picosecond and at the end of the pipeline, adaptation can take seconds and color naming minutes.

An important feature were the bidirectional arrows: we have a feedback loop with control moving down and information moving up. Because of the sequence of recoding and the feedback, the receptors in the retina are not like pixels in a CCD sensor

• Receptive field: area of visual field that activates a retinal ganglion (H.K. Hartline, 1938)
• Center-surround fields allow for adaptive coding (transmit contrast instead of absolute values)
• Horizontal cells presumed to inhibit either its bipolar cell or the receptors: opponent response in red–green and yellow–blue potentials (G. Svaetichin, 1956)
• Retinal ganglion can be tonic or phasic: pathway may also be organized by information density or bandwidth

The last item comes from a table of the parvocellular and magnocellular pathways Lucia Rositani-Ronchi compiled for me at the 1993 AIC meeting in Budapest:

	P–	M–
Originating retinal ganglion cells	Tonic	Phasic
Temporal resolution	Slow (sustained responses, low conduction velocity)	Fast (mostly transient responses, some sustained, high conduction velocity)
Modulation dominance	Chromatic	Luminance
	Adaptation occurs at high frequencies	Adaptation occurs at all frequencies
Color	Receives mostly opponent type input from cones sensitive to short and long wavelengths	Receives mostly combined (broadband) input from M and L cones, both from the center and from the surround of receptive fields
Contrast sensitivity	Low (threshold > 10%)	High (threshold < 2%)
LGN cell saturation	Linear up to about 64% contrast	At 10%
Spatial resolution	High (small cells)	Low (large cells)
Spatio-temporal resolution	When fixation is strictly foveal, extraction of high spatial frequency information (test gratings), reflecting small color receptive fields	Responds to flicker
	Long integration time	Short integration time
Relation to channels	Could be a site for both a lightness channel as for opponent-color channels. The role depends on the spatio-temporal content of the target used in the experiment	Might be a site for achromatic channels because the spectral sensitivity is similar to Vλ, it is more sensitive to flicker, and has only a weak opponent color component
Possible main role in the visual system	Sustain the perception of color, texture, shape, and fine stereopsis	Sustain the detection of movement, depth, and flicker; reading of text

We have four retinal pigments (erythrolabe, chlorolabe, cyanolabe, rhodopsin) attached by a lysine to a protein backbone. These four pigments are sensitized to photons at 4 energy levels (wavelengths): L, M, S, and rods. The energy levels are not numbers but distributions, namely the probabilities for a photon catch with that chromatophore.

A 3-dimensional signal with L, M, S is not efficient because we need a high spatial resolution but the chromatic information can be at a lower resolution. This is reflected in the modulations transfer functions for the HVS and is exploited for example in image encoding, where we transform an RGB signal into a color opponent signal and then down-sample the chroma images:

In 1993, it was not known where this transformation occurs in the HVS. In fact, there is quite a bit of processing in the retina, and many details are still unknown.

In their recent paper The elementary representation of spatial and color vision in the human retina, Ramkumar Sabesan et al. report on a seminal study to locate where in the HVS, the luma-chroma encoding occurs in the parvocellular pathway.

Using an adaptive optics scanning laser ophthalmoscope (AO-SLO), the authors studied 174 L-cone, 99 M-cone, and 12 S-cone samples by stimulating them individually with a 543 nm, 500 ms pulse and asking two subjects to report the perceived color name. The names were restricted to red, green, blue, yellow, white, and not seen.

The subjects reported achromatic sensations 61.8% of the time. When red was reported (22.5% of seen trials), it was more likely to be driven by L- than M-cones, whereas green (15.7%) was more likely to come from the excitation of M-cones. Thus, L-cones tended to signal both white and red, whereas M-cones tended to signal both white and green. The observation that these color percepts roughly align with the predictions of large-field cone-isolating stimuli suggests that the same opponent neuronal circuits may be implicated in both paradigms. This finding also supports the idea that the visual system can learn the spectral identity of individual cones.

The apparent segregation of color categories into distinct populations of cells is suggestive of a parallel representation of color and achromatic sensations. Moreover, these results imply that, for a large number of cones, their individual activation is not sufficient to produce a color. (Remember that in this experiment single cones are excited; in free vision, most cones are activated and the eye saccades, presenting a point in the visual field to several cones.)

The authors found that the cones most likely to generate strong spectral opponency in a parvocellular neuron, that is, those surrounded by cones of opposing type, were not more likely to generate red or green percepts. Rather, all these examples, when stimulated in isolation, drove achromatic percepts on a majority of the trials.

There is little doubt that the long-duration supra-threshold stimulation of individual cones here influences the firing of a number of different ganglion cell types. In particular, a multi-electrode study demonstrated that the activation of a single cone simultaneously evoked responses in both midget (parvocellular) and parasol (magnocellular) ganglion cells. The results may be particularly informative in differentiating proposals about the role of parvocellular neurons in achromatic spatial and color vision.

The study confirms the old result that the red-green system samples the visual world at a lower resolution than the achromatic system. The new results from the studies reported in the present paper are consistent with the idea that the HVS represents these two pieces of information with separate pathways that emerge as early as the photoreceptor synapse: one chiefly concerned with high-resolution achromatic vision and a second, lower-resolution color system.

The luma-chroma transformation with chroma subsampling is very important in image processing. In your opinion, does this new result allow the design of better imaging pipelines? Does this allow us to design better retinex algorithms? Join the conversation in the Trellis group.

Citation: R. Sabesan, B. P. Schmidt, W. S. Tuten, A. Roorda, The elementary representation of spatial and color vision in the human retina. Sci. Adv. 2, e1600797 (2016).

Retina from iPS cells

On 29 July 2016, the Japan News by the Yomiuri Shinbun reported that the transplant of iPS cells has been approved in Kobe.

The ethics committee of the Kobe City Medical Center General Hospital has approved a surgery plan to transplant retina developed from donor-derived induced pluripotent stem (iPS) cells to an eye-disease patient. The hospital in Kobe will aim to carry out the surgery, part of the world’s first clinical study of iPS cells, in the first half of 2017, after getting the green light from the government.

The subject of the treatment will be a patient with age-related macular degeneration, a serious disease that can lead to blindness. The Riken Center for Developmental Biology, or CDB, will generate retina from iPS cells, supplied by Kyoto University’s Center for iPS Cell Research and Application, from donors with no blood ties to the patient.

Link to the article

Retina's transcripotme

From Santiago Ramón y Cajal 's time we have known that there are five types of neuronal cells in the retina: rods & cones, horizontal cells, bipolar cells, amacrine cells, and retinal ganglion cells.

With Sharpe et al. (L. T. Sharpe, A. Stockman, H. Jgle, and J. Nathans. opsin genes, cone photopigments and color vision. Color vision: From genes to perception, pages 3–51, 1999) we learned that the spectral sensitivity of the pigments in the cones is controlled by the not-so-robust order of the visual pigment genes in the sex chromosome and color vision deficiency had to do with the L peak moving towards the M peak or vice versa.

However, the genome only allowed to predict a predisposition for color vision deficiency, not a prediction of the spectral color performance. The reason is that not the genes determine the spectral peaks but their expression by the transcriptome, i.e., the messenger RNA (mRNA), which of course cannot be studied in vivo.

In a recent paper (E. Z. Macosko, A. Basu, R. Satija, J. Nemesh, K. Shekhar, M. Goldman, I. Tirosh, A. R. Bialas, N. Kamitaki, E. M. Martersteck, et al. Highly parallel genome-wide expression profiling of individual cells using nanoliter droplets. Cell, 161(5):1202–1214, 2015), Macosko et al. describe the application of a new technique called Drop-seq, which has allowed them to analyze the gene activity of 44,808 cells from 14-day-old mice retinae. "Gene activity" here means that they analyzed the transcriptomes of these 44,808 retinal cells and identified 39 transcriptionally distinct cell populations, each corresponding to one of a group of closely related cell types.

Drop-seq generates a library of STAMPs (single-cell transcriptomes attached to micro-particles). They used Seurat, a recently developed R package for single-cell analysis, to study this STAMP library. In a first step, they performed a principal component analysts on the largest libraries, then they reduced the 32 statistically significant principal components to two dimensions using t-distributed stochastic neighbor embedding (tSNE).

Subsequently they projected the remaining cells in the data into the tSNE analysis. Then they combined a density clustering approach with post hoc differential expression analysis to divide the 44,808 cells among 39 transcriptionally distinct clusters, obtaining this illustration:

Finally, they organized the 39 cell populations into larger categories (classes) by building a dendrogram of similarity relationships among the 39 cell populations. For now, the result is that they can say a lot about the amacrine cells that was not known before. However, it will take more research to formulate an interpretation for the visual system.

International Year of Light

Here in Switzerland the weather tends to be bad and we have a Zwinglian/Calvinistic Leitkultur, which might explain our tendency towards pessimism and feeling more unlucky than lucky: it is customary to first look at the negative side of things and then to let us be surprised and feel lucky when things turn out to be positive. In this context, nobody is surprised when the newspapers announce the new year by listing negative anniversaries: 700 years Morgarten, 500 years Marignano, 70 years end of World War II.

Today's Zürich is home to many computer science labs and the city has as many nerds as gnomes (the equivalent persona in banking). They may see 2015 as the year of the palindrome, because 2015₁₀ = 11111011111₂. Or the many mathematicians in Zürich will see 2015 as a Japanese cube, because in the Japanese calendar it is 平成２７年 or Heisei 27 = 3³. For movie buffs, this year is MMXV.

For color scientists, 2015 is the International Year of Light and Light-based Technologies, a United Nation observance that aims to raise awareness of the achievements of light science and its applications, and its importance to humankind. The IYL 2015 will launch at the UNESCO headquarters in Paris on 19 January 2015, with the unveiling of 1001 Inventions and the World of Ibn Al-Haytham.

Indeed, 2015 marks the anniversaries of several events related to light, optics, and vision:

• 1015, a millennium ago, the Iraqi scientist Ibn Al-Haytham published his Book of Optics
• 1815 Augustin-Jean Fresnel proposed the notion of light as a wave
• 1865 James Clerk Maxwell proposed the electromagnetic theory of light propagation
• 1915 Albert Einstein embedded his 1905 theory of the photoelectric effect into cosmology through general relativity
• 1965 Arno Penzias and Robert Woodrow Wilson discovered the cosmic microwave background
• 1965 Charles Kao theorized and proposed to use glass fibers to implement optical broadband communication

In ancient Greece, there where two competing theories of vision. One theory was called the emission theory (Euclid, Ptolemy) and claimed that vision worked by little flame exiting the eye, traveling on rays, scanning the objects in the visual field, and traveling back to the eye reporting what they detected. In the intromission theory (Aristotle), when an object is looked at, it replicates itself and the replica travels along a ray into the viewer's eye, where it is seen.

For a millennium, there was a raging discussion of whether the emission theory or the intromission theory was the correct one. This discussion was based purely on theoretical considerations and heuristics. In his 1015 book, Ibn Al-Haytham introduced the modern concept of scientific research based on experimentation and controlled testing that we still use today: a hypothesis is formulated, an experiment is conducted varying the parameters, the results of the experiment are discussed, and the conclusions are drawn. Because of this, Ibn Al-Haytham is often referred to as the first scientist.

Using the scientific method, Ibn Al-Haytham developed the first plausible theory of vision. Among other contributions, he also explained the camera obscura and catoptrics. He has strongly influenced later scientists like Averroes, Leonardo da Vinci, Galileo Galilei, Christian Huygens, René Descartes, and Johannes Kepler.

Ibn Al-Haytham's full name was Abū ʿAlī al-Ḥasan ibn al-Ḥasan ibn al-Haytham. His Latinized name was originally Alhacen; since 1572, when Friedrich Risner misspelled his name, in the West he has been known as Alhazen. He was born and raised in Basra, where he initially worked. Later he worked in Baghdad and Cairo.

For more information on the International Year of Light see here.

Glistenings in pseudophakic vision.

As we age, our crystalline lens becomes cloudy and we call it a cataract, maybe because the world is seen as from behind a large foaming waterfall. Already the Romans carried out cataract operations 2000 years ago, so the medical remedy is pretty much routine: the cataract is removed surgically and replaced with an intra-ocular lens (IOL). Such an IOL is generally known as a pseudophakic IOL and provides the light focusing function originally undertaken by the crystalline lens.

The ancient Greek word for lens is phakos, so phakia is the presence of the natural crystalline lens. Pseudophakia is the substitution of the natural crystalline lens with an IOL.

One problem of the pseudophakic patient is that sometimes in the weeks or months after the surgical procedure, visual discomfort due to glistenings is experienced. The glistenings are due to micro-vacuoles in the IOL, the vacuoles being part of the polymer's structure. After the IOL has been implanted, water can fill these vacuoles and water has a different refraction index (1.33) than the polymer ~1.55).

What was not known is how much the glistenings impact visual performance. One research technique was to measure the MTF. However, although MTF is related to visual acuity, it is not related to global contrast. and does not explain the visual discomfort. Alessandro Franchini, Andrea Romolo and Iacopo Franchini implemented a ray-tracing program to model and analyze the effect of the vacuoles.

They found that when a light source is in the field of view, without glistenings a clear secondary image is produced, but with glistenings light scattering introduces noise on entire visual field, reducing the global contrast.

The solution is to use hydrophobic acrylic lenses and to keep them in water before implanting them. With this, the IOL will contain 4% water instead of the usual 2%. After the lens is implanted, there will not be the current of liquids that causes the glistenings.

Citation: Alessandro Franchini, Andrea Romolo and Iacopo Franchini, Effect of glistenings on the pseudophakic patient vision, Atti della Fondazione Giorgio Ronchi, Vol. LXIX, N. 5, pp. 589–599.

Towards a cure for macular degeneration

Japanese researchers say they have conducted the world's first surgery using iPS cells, on a patient with a macular degeneration. The operation is seen as a major step forward in regenerative medicine. A team led by Masayo Takahashi from a RIKEN research lab in Kobe performed the operation on Friday with the cooperation of a team from the Institute of Biomedical Research and Innovation.

The patient was a woman in her 70s with age-related macular degeneration, which involves a progressive decline in vision. The researchers obtained a small amount of the patient's skin cells and turned them into induced pluripotent stem cells. Using the iPS cells' ability to develop into any kind of body tissue, the team then transformed them into retinal tissue.

Part of the patient's deteriorating retina was then surgically replaced with the iPS-derived tissue. The patient reportedly came out of anesthesia after being under for approximately 3 hours. The researchers told reporters the patient is recovering well in a hospital room. They added there has been no excessive bleeding or other problems.

Yasuo Kurimoto of the Institute of Biomedical Research and Innovation said he believes the surgery was successful. Masayo Takahashi of the RIKEN lab said she's relieved the surgery was completed safely. She added that although she wants to believe the first clinical case was a major step forward, much more development is needed to establish iPS surgery as a treatment method.

The researchers say the primary objective of the operation was to check the safety of the therapy. They say that since the patient has already lost most of her vision-related cells, the retinal transplant would only slightly improve her eyesight or slow its loss. But the researchers say the therapy could become a fundamental cure if its safety and efficacy can be confirmed by the transplant.

They plan to monitor the patient over the next 4 years. iPS cells were developed by Kyoto University Professor Shinya Yamanaka, who was awarded the 2012 Nobel Prize in Physiology or Medicine. This first-ever use of such cells in a human patient is seen as a major step forward for regenerative medicine — a kind of therapy aimed at restoring diseased organs and tissue.

Source: http://www3.nhk.or.jp/nhkworld/english/news/20140912_53.html

You only see what you want to see

Scientists sometimes have funny ways to name entities. Laymen then do not know if the topic is serious or their leg is being pulled. For example, in high energy physics the types of quarks are called flavors, and the flavors are called up, down, strange, charm, bottom, and top.

Molecular biologists tend to have even more bizarre ways to name their entities. For example, in wet color science to study top-down modulation in the visual system they may breed loxP-flanked tdTomato reporter mice with parvalbumin-, somatostatin-, or vasoactive intestinal peptide-Cre positive interneuron mice.

But then, these wet experiments in physiological research are very difficult and tedious. In practical color science, we mostly take a bottom-up approach, which most of the time works acceptably in engineering terms, but then can fail miserably in corner cases. More complete models are possible only when we take into account top-down processes, because in the visual system most information is transmitted top-down, not bottom-up.

Building models is a creative process in which one can easily get carried away, so in color science we have always to question the physiological basis for each model we propose. It is this physiological research that is very difficult. Recently a team from the University of California, Berkeley and Stanford University here in Palo Alto (Siyu Zhang, Min Xu, Tsukasa Kamigaki, Johnny Phong Hoang Do, Wei-Cheng Chang, Sean Jenvay, Kazunari Miyamichi, Liqun Luo and Yang Dan) have accomplished such a feat.

We often focus on a particular item out of a thousand objects in a visual scene. This ability is called selective attention. Selective attention enhances the responses of sensory nerve cells to whatever is being observed and dampens responses to any distractions. Zhang et al. identified a region of the mouse forebrain that modulates responses in the visual cortex. This modulation improved the mouse's performance in a visual task.

Before the work of Zhang et al., the synaptic circuits mediating top-down modulation were largely unknown. Among others, because long-range corticocortical projections are primarily glutamatergic, whether and how they provide center-surround modulation was unknown.

To examine the circuit mechanism of top-down modulation in mouse brain, Zhang et al. first identified neurons in the frontal cortex that directly project to visual cortex by injecting fluorescent latex microspheres (Retrobeads) into V1. They found numerous retrogradely labeled neurons in the cingulate area. To visualize the axonal projections from cingulate excitatory neurons, they injected adeno-associated virus [AAV-CaMKIIα-hChR2(H134R)-EYFP] into the cingulate.

They discovered that somatostatin-positive neurons strongly inhibit pyramidal neurons in response to cingulate input 200 μm away. That they also mediate suppression by visual stimuli outside of the receptive field suggests that both bottom-up visual processing and top-down attentional modulation use a common mechanism for surround suppression.

Citation and link: Long-range and local circuits for top-down modulation of visual cortex processing. Siyu Zhang, Min Xu, Tsukasa Kamigaki, Johnny Phong Hoang Do, Wei-Cheng Chang, Sean Jenvay, Kazunari Miyamichi, Liqun Luo, and Yang Dan Science 8 August 2014: 345 (6197), 660-665. [DOI:10.1126/science.1254126]

Photon Hunting in the Twilight Zone

Deep in the twilight zone of the ocean, small, glowing sharks have evolved special eye features to maximize the amount of light they see, researchers report this week in PLOS ONE. The scientists mapped the eye shape, structure, and retina cells of five deep-sea bioluminescent sharks, predators that live 200 to 1000 meters deep in the ocean, where light hardly penetrates.

The sharks have developed many coping strategies. Their eyes possess a higher density of rods than those of nonbioluminescent sharks, which might enable them to see fast-changing light patterns. Such ability would be particularly useful when the animals emit light to communicate with one another. Some species also have a gap between the lens and the iris to allow extra light in the retina, a feature previously unknown in sharks.

In the eyes of lantern sharks (Etmopteridae), the scientists discovered a translucent area in the upper socket. The researchers suspect this feature might help the sharks adjust their glow to match the sunlight for camouflage.

I wonder if we computer nerds will evolve our visual system similarly.

Citation (Open Access):

Claes JM, Partridge JC, Hart NS, Garza-Gisholt E, Ho H-C, et al. (2014) Photon Hunting in the Twilight Zone: Visual Features of Mesopelagic Bioluminescent Sharks. PLoS ONE 9(8): e104213. doi:10.1371/journal.pone.0104213

Tic-tac-toe patent 8,770,625 in color

As noted on lines 23 and 24 in column 4 of the printed version of patent 8,770,625,
the U.S. Patent Office procedure discourages the use of color drawings. This makes Fig. 4 a little hard to visualize for the non color scientist (there are no color figures in Wyszecki & Stiles), so here it is in color (right pane):

The invention is relatively simple. The general field is anti-counterfeiting as it applies to packaging. Professional counterfeiters have no problem faking ordinary measures like serial numbers and holograms, so the trick is to embed information that cannot easily be perceived by a counterfeiter, hence is omitted in the facsimile. Fortunately color does not exist in nature, it is just an illusion happening in our minds. Therefore, all we have to do is to create an illusion you can only perceive if you expect it.

As described in patent 8,770,625, a number computed from the—possibly counterfeited—serial number on the package can be encoded positionally in a tic-tac-toe grid. The marking is just above the visual threshold, so the naive counterfeiter will reproduce the same pattern on all packages. The trained inspector can then quickly assert whether an actual positional code corresponds, for example, to the possibly fake serial number.

Patent 8,770,625 is relatively short with just three claims, but reducing it to practice is a little tricky, even when all the steps are disclosed in the patent. The difficult part is to design the tool to determine experimentally the visual thresholds for the print process being used and the light conditions under which the inspections are expected to happen. You need to be skilled in the art.

The above figure is a screen-shot of that tool. To implement it you need to write a spectral color management system with CIE colorimetry to simulate the press on the display and vision colorimetry to model what the actual human visual system perceives. The details of the controls are explained in patent 8,770,625.

Depending on your viewing conditions, the above color version of Fig. 4 might be under the visual threshold. If that is the case, in the figure below we crank up the saliency and decrease the background coverage, so you will see the encoding for sure. If you have aliasing problems, you can click on the figures to display them at the original resolution in which they were created eight years ago, early July 2006 (time flies).

Staring at computers all day alters your eyes

As a color scientist you already know that you have to position your display and chair combination so that the top bezel is at the same height as your eyes. The reason is so your eyes are not wide open and dry out. You also avoid sticking a personal fan in the display's USB port and tilt the display face slightly down so you cannot see light fixture reflections.

To my surprise, although this is usually explained in the ergonomics booklets shipping with computers, this is not generally known and scientists can still get research grants to study it (The Osaka Study):

The data obtained in the present study suggest that office workers with prolonged VDT (visual display terminal) use, as well as those with an increased frequency of eye strain, have a low MUC5AC (mucin 5AC) concentration in their tears. Furthermore, MUC5AC concentration in the tears of patients with DED (dry eye disease) may be lower than that in individuals without DED.

Citation: Uchino Y, Uchino M, Yokoi N, et al. Alteration of Tear Mucin 5AC in Office Workers Using Visual Display Terminals: The Osaka Study. JAMA Ophthalmol. Published online June 05, 2014. doi:10.1001/jamaophthalmol.2014.1008.

The paper costs $30, but you can read the current JAMA issue for free if you register.

A new class of bipolar cells

It took 225 undergraduates more than 20,000 hours of work to map the wiring diagram of a 117 µm by 80 µm patch of a mouse retina. They did discover a new class of bipolar cells, however the patch was too small to determine its exact function: a larger patch is necessary. They will try to achieve this through a crowd-sourcing project known as EyeWire.

News article: Making connections in the eye