9,355,339: system and method for color reproduction tolerances

In the early days of digital imaging for office use, there was a religion that the response of the system had to be completely neutral. The motivation came from a military specification that a seventh generation copy had to still look good and be readable. If there would be a deviation from neutrality, every generation would enhance this deviation.

When a Japanese manufacturer entered the color copier market, they enhanced the images to improve memory colors and boost the overall contrast. They were not selling to the government and rightfully noted that commercial users do not do multiple generation copies.

Color is not a physical phenomenon, it is an illusion and color imaging is about predicting illusions. Visit your art gallery and look carefully at an original Rembrandt van Rijn painting. The perceived dynamic range is considerably larger than the gamut of the paints he used because he distorted the colors depending on their semantics. With Michel Eugène Chevreul's discovery of simultaneous contrast, the impressionists then went wild.

When years later I interviewed for a position in a prestigious lab, I gave a talk on how—based on my knowledge of the human visual system (HVS)—I could implement a number of algorithms that greatly enhance the images printed on consumer and office printers. The hiring engineering managers thought I was out of my mind and I did not get the job. For them, the holy grail was the perfect neutral transmission function.

This neutral color reproduction is a figment of imagination in the engineer's minds in the early days of digital color reproduction. The scientists who earlier invented the mechanical color reproduction did not have this hang-up. As Evans observed on page 599 of [R.M. Evans, “Visual Processes and Color Photography,” JOSA, Vol. 33, 11, pp. 579–614, November 1943], under the illuminator condition color constancy mechanisms in the human visual system (HVS) correct for improper color balance. As Evans further notes on page 596, we tend to remember colors rather than to look at them closely; for the most part, he notes, careful observation of stimuli is made only by trained observers. Evans concludes that it is seldom necessary to obtain an exact color reproduction of a scene to obtain a satisfying picture, although it is necessary that the reproduction shall not violate the principle that the scene could have thus appeared.

We can interpret Evans’ consistency principle (page 600) as what is important is the relation among the colors in a reproduced image, not their absolute colorimetry. A color reproduction system must preserve the integrity of the relation among the colors in the palette. In practice, this suggests that three conditions should be met. The first is that the order in a critical color scale should not have transpositions, the second is that a color should not cross a name boundary, the third is that the field of reproduction error vectors of all colors should be divergence-free. The intuition for the divergence condition is that no virtual light source is introduced, thus supporting color constancy mechanisms in the HVS.

We all know the Farnsworth-Munsell 100 hue test. The underlying idea is that when an observer has poor color discrimination, either due to a color vision deficiency or due to lack of practice, this observer will not be able to sort 100 specimen varying only in hue. In the test, the number of hue transpositions is counted to score an observer.

We can considerably reduce the complexity of a color reproduction system if we focus on the colors that actually are important in the specific images being reproduced. We can minimally choose the quality of the colorants, paper, halftoning algorithm, and the number of colorants to just preserve the consistency of that restricted palette.

First, we determine the palette, then we reproduce a color scale with a selection of the above parameters and we give the resulting color scale image a partial Farnsworth-Munsell hue test. This is the idea behind invention 9,355,339. The non-obvious step is how to select color scales and how to use a color management system to simulate a reproduction. The whole process is automated.

A graphic artist manually identifies the locus of the colors of interest, which is different for each application domain, like reproductions of cosmetics or denim clothes. A scale encompassing these critical colors is built and printed in the margin. In the physical application, the colors are measured and the transpositions are counted.

The more interesting case is a simulation. There is no printing, just the application of halftoning algorithms and color profiles. The system extracts the final color scale and identifies the transpositions compared to the original. Why is the simulation more important?

Commercial printing is a manufacturing process and workflow design is crucial to the financial viability of the factory. Printing is a mass-customization process, thus every day in the shop is different. This makes it impossible to assume a standard workload.

For each manufacturing task, there are different options using different technologies. For example, a large display piece can be printed on a wide bubblejet printer, or the fixed background can be silkscreen printed and the variable text can be added on the wide bubblejet printer. Another example is that a booklet can be saddle stitched, perfect bound, or spiral bound.

In an ideal case, a print shop floor can be designed in a way to support such reconfigurable fulfillment workflows, as shown in this drawing (omitted here are the buffer storage areas).

However, this would not be commercially viable. In manufacturing, the main cost item is labor. Different fulfillment steps require different skill levels (at different salary levels) and take a different amount of time.

Additionally, not all print jobs are fulfilled at the same speed. A rush job is much more profitable than a job with a flexible delivery time. To stay in business, the plant manager must be able to accommodate as many rush jobs as possible.

This planning job is similar to that of scheduling trains in a saturated network like the New Railway Link through the Alps (NRLA or NEAT for Neue Eisenbahn-Alpentransversale, or simply AlpTransit). For freight, it connects the ports of Genoa and Rotterdam, with an additional interchange in Basle (the Rotterdam–Basel–Genoa corridor). For passengers, it connects Milan to Zurich and beyond. There are two separate basis tunnel systems: the older Lötschberg axis and the newer Gotthard axis. In the latter, freight trains travel at least at 100 km/h and passenger trains at least at 200 km/h. These are the operational speeds, the maximal speed is 249 km/h, limited by the available power supply.

The trains are mostly of the freight type, traveling at the lower operational speed. A smaller number of passenger trains travels at twice the operational speed. A freight train can be a little late, but a passenger train must always be on time lest people miss their connections, which is not acceptable.

The trains must be scheduled so that passenger trains can pass freight trains when these are at a station and can move to a bypass rail. However, there can be unforeseen events like accidents, natural disasters, and strikes that can delay the trains on their route. The manager will counter these problems by accelerating trains already in the system above their operational speed when there is sufficient electric power to do so. This way, when the problem is solved, there is additional capacity available.

The problem with modifying the schedule is that one cannot just accelerate trains: new bypass stations have to be determined and the travel speeds have to be fine-tuned. Train systems are an early implementation of industry 4.0 because the trains also automatically communicate between each other to avoid collisions and to optimize rail usage. For AlpTransit this required solving the political problem of forcing all European countries to adopt a new ERTMS/ETCS (European Train Control System) Level 2, to which older locomotives cannot be upgraded.

The regular jobs and rush jobs in a print fulfillment plant are similar. The big difference is that the train schedule is the same every day, while in printing each day is completely different. The job of the plant designer is to predict the bottlenecks and do a cost analysis to alleviate these bottlenecks. In particular, deadlocks have to be identified and mitigated. There are two main parameters: the number and speed of equipment, and the amount of buffer space. Buffering is highly nonlinear and cannot be estimated by eye or from experience. The only solution is to build a model and then simulate it.

We used Ptolemy II for the simulation framework and wrote Java actors for each manufacturing step. To find and mitigate the bottlenecks, but especially to find the dreaded deadlock conditions, we just need to code timing information in the Java actors and run Montecarlo simulations.

We used a compute cluster with the data encrypted at rest and using Ganymed SSH-2 for authentication with certificates and encryption on the wire. Each actor could run on a separate machine in the cluster. The system allows the well-dimensioned design of the plant, its enhancement through modernization and expansion, and the daily scheduling.

So far, the optimization is just based on time. In a print fulfillment plant, there are also frequent mistakes in the workflow definition. The workflow for a job is stored in a so-called ticket (believe me, reaching a consensus standard was more difficult than for ERTMS/ETCS). One of the highest costs in a plant is an error in the ticket, which causes the job to be repeated after the ticket has been amended. With this, risk mitigation through ticket verification is a highly valuable function, because it allows a considerable cost reduction for not having to allocate insurance expenses.

While in office printing A4 or letter size paper are the norm, commercial printers use a possibly large paper size to save printing time and, with it, cost. This means there are ganging and imposition, folding, rotating, cutting, etc. It is easy to make an imposition mistake and pages end up in the wrong document or at the wrong place. Similarly, paper can be cut at the wrong point in the process or folded incorrectly.

Once we have a simulation of the print fulfillment factory, we can easily solve these workflow problems, thus reducing risk and with it insurance cost. The data for print jobs is stored in portable document format (PDF) files. For each workstation in a print fulfillment plant, we take the Java actor implemented for the simulation and add input and output ports for PDF files. We then implement a PDF transformer for each workstation that applies the work step to the PDFs. There can be multiple input and output PDFs. For example, a ganging workstation takes several PDF files and outputs a new PDF file for each imposed sheet.

Most errors happen when a ticket is compiled. After simulating the workflow, the operator simply checks the resulting PDF files. A mistake is immediately visible and can be diagnosed by looking at the intermediate PDF files. A more subtle error source is when workstations negotiate workflow changes in the sense of the industry 4.0 technology. Before the change can be approved, the workflow has to be simulated again and the difference between the two output PDF files has to be computed.

A more valuable, but also more complex workflow change is accommodating rush jobs by taking shortcuts. For example, if there is a spot color in the press, can we reuse the same color or do we have to clean out the machine to change or remove the spot color? Another example is the question of using dithering instead of stochastic halftoning to expedite a job. Finally, earlier we mentioned the possibility of running a fixed background through a silkscreen press and printing only the variable contents on a bubblejet press.

In a conventional setting, any such change requires doing a new press-check and having the customer come in to approve it. In practice, this is not always realistic and the owner will use his best judgment so self-approve the proof.

9,355,339 automates this check and approval. The ICC profiles are available and can be used to compute the perceived colors in each case. The transposition score for the color scale (there can be more than one) can predict the customer's approval of the press-check.

Thus, we have created a Java actor that simulates a human and can predict the human's perception. This is an industry 4.0 application where we not only have semantic models for the machines but also for the humans, and the machines can take the humans into consideration.

9,098,487: categorization based on word distance

One of the correlates for the social appreciation or value of scientists is the Gini coefficient. Indeed, poor people cannot afford technologies that make their lives more comfortable (and would not be able to amortize the investment anyway because their labor has little value). Rich people also cannot necessarily amortize the investment in new technology, because they can just hire the poor to do the work for them for a low pay. What drives technology is a low Gini coefficient, because it is a broad middle class that can and does invest in new technologies that makes their lives more efficient.

Worldwide, over the past two centuries the Gini factor has been on the rise, yet there has been incredible technological progress. This means we have to look at a smaller geographical scale. Indeed, for the late 2000s, the United States had the 4th highest measure of income inequality out of the 34 OECD countries measured, after taxes and transfers had been taken into account. As it happens, except for the medical sciences, the American science establishment is only a pale shadow of what it was half a century ago.

In this context, we were naive when three years ago we embarked to solve an efficiency problem for customers processing medical bills, invoices and receipts. These artifacts are still mostly on paper and companies spend a huge amount of time having minimum wage people ingesting it for processing in their accounting systems. In the datasets we received from the customers, the medical bills were usually in good physical shape, but the spelling checkers in the OCR engines had a hard time cracking the cryptic jargon. Receipts were in worst physical shape, printed on poor printers and crumpled, and often imaged by taking a picture with a smart phone.

Surprisingly, invoices were also difficult to process. They often had beverage stains and sometimes they had been faxed several times using machines that looked like they had mustard on the platen and mayonnaise under the lid. But the worst was that in the datasets we received, the form design of the invoices was very inconsistent: the fields and their labels were all over the place, many gratuitous and inconsistent abbreviations were used, etc.

Considerable effort had been spent in the 80s to solve this problem by people like Dan Bloomberg with his mathematical morphology methods to clean up the scanned images (Meg Withgott coined the phrase "document dry cleaning") and Rob Tow with David Hecht and others invented the glyph technology to mark forms so that the location and semantics of each field could be looked up. Maybe due to the increasing Gini coefficient they were not commercially successful. However because this time we had actual customers, we decided to give it another go.

Note that OCR packages already have pretty good built-in spelling checkers, so we are dealing with hard cases. The standard approach used in semantic analysis is based on predetermining all situations for a lexeme and store them into NoSQL databases. In our applications this turned out to be too slow: we needed a response time under 16 µs.

Looking at our dataset, we had a number of different issue types:

• synonym: a word or phrase that means exactly or nearly the same as another word or phrase in the same language
• abbreviation: a shortened form of a word or phrase
• figure of speech: a word or phrase used in a nonliteral sense to add rhetorical force to written passage
• misspelling: a word or phrase spelled incorrectly
• mistyping: like misspelling, but depends on the distance between the keys on a keyboard or the stroke classes in the OCR engine
• metonym: a word, name, or expression used as a substitute for something else with which it is closely associated
• synecdoche: a figure of speech in which a part is made to represent the whole or vice versa
• metalepsis: a figure of speech in which a word or phrase from figurative speech is used in a new context
• kenning: a compound expression in Old English and Old Norse poetry with metaphorical meaning
• acronym: an abbreviation formed from the initial letters of other words and pronounced as a word

In addition to computational speed, we wanted to be able to address each of these constructs explicitely. In the following we call a lexeme "category" because the used process is a categorization more than a linguistic analysis.

We solved the problem by introducing a metric, so that we could deal with everything as distances and intervals. For the metric we chose the edit distance, also known as Levenshtein distance: the number of edits used to change a first word in the category into a second word in the category, such as using additions, deletions, substitutions, and transpositions. This metric can be computed very fast. We also tried the Damerau-Levenshtein distance, but in this application it did not make a difference.

With a metric, everything was simple. We took the lexemes in each category and computed the center of gravity to determine the prototype for each category and the diameter of the category. The category then received a label that is the word or phrase used in the document typing application.

With misspellings the diameters could be small, but with with synonyms and figures of speech the intervals could be quite large and can overlap. The intersections could easily be computed. Because in all three out datasets the dictionaries were small, we easily resolved the overlaps visually by constructing a graph in which each node had the category diameter as the value and the edges between two nodes had their Levenshtein distance as a weight. Then we plotted the graph with Gephi and split up the overlapping categories into smaller one with the same label.

With this, document typing became very fast: for each word or phrase we looked into which category it fell, and in there we looked for a match. When there was one, we replaced it with the lexeme's label, when not, we added it to the category and logged it for manual verification.

Patent 9,098,487 was filed November 29, 2012 and issued August 4, 2015.

Tic-tac-toe patent 8,770,625 in color

As noted on lines 23 and 24 in column 4 of the printed version of patent 8,770,625,
the U.S. Patent Office procedure discourages the use of color drawings. This makes Fig. 4 a little hard to visualize for the non color scientist (there are no color figures in Wyszecki & Stiles), so here it is in color (right pane):

The invention is relatively simple. The general field is anti-counterfeiting as it applies to packaging. Professional counterfeiters have no problem faking ordinary measures like serial numbers and holograms, so the trick is to embed information that cannot easily be perceived by a counterfeiter, hence is omitted in the facsimile. Fortunately color does not exist in nature, it is just an illusion happening in our minds. Therefore, all we have to do is to create an illusion you can only perceive if you expect it.

As described in patent 8,770,625, a number computed from the—possibly counterfeited—serial number on the package can be encoded positionally in a tic-tac-toe grid. The marking is just above the visual threshold, so the naive counterfeiter will reproduce the same pattern on all packages. The trained inspector can then quickly assert whether an actual positional code corresponds, for example, to the possibly fake serial number.

Patent 8,770,625 is relatively short with just three claims, but reducing it to practice is a little tricky, even when all the steps are disclosed in the patent. The difficult part is to design the tool to determine experimentally the visual thresholds for the print process being used and the light conditions under which the inspections are expected to happen. You need to be skilled in the art.

The above figure is a screen-shot of that tool. To implement it you need to write a spectral color management system with CIE colorimetry to simulate the press on the display and vision colorimetry to model what the actual human visual system perceives. The details of the controls are explained in patent 8,770,625.

Depending on your viewing conditions, the above color version of Fig. 4 might be under the visual threshold. If that is the case, in the figure below we crank up the saliency and decrease the background coverage, so you will see the encoding for sure. If you have aliasing problems, you can click on the figures to display them at the original resolution in which they were created eight years ago, early July 2006 (time flies).

White House tackles trolls

On June 4 the White House issued a fact sheet that laid out a framework for five recommended executive actions and seven legislative recommendations for both branches of government to address as a means to tackle "patent trolls." For example, the Patent and Trademark Office will draft a proposal to require patent applicants and owners to "regularly update ownership information" as a means of protecting against the creation of shell companies for hiding abusive litigants. Further, it will permit more discretion in awarding fees to prevailing parties in patent cases, providing district courts with more discretion to award attorney’s fees as a sanction for abusive court filings. It also calls on Congress to craft similar legislative language.

Link: Fact Sheet: White House Task Force on High-Tech Patent Issues

Proofing a page for color discriminability problems based on color names

Now that there is a physiological basis for color categorization, we can ask ourselves what this is good for. We cannot eat it, but it might have considerable commercial value in United States Patent 8,456,694 issued two days ago on June 4, 2013. I am not a lawyer, but it appears that if you take two colors, determine their names, and then do anything with it, you might have to license this patent (but you can still keep your hippocampus ☺).

In large American corporations, when a new CEO start their new position, they often begin by putting their mark on the company's branding. They remodel their office and maybe even the HQ entrance, tweak the logo, design a new font, change the corporate palette, etc. These endeavors cost millions of dollars, but big corporations can afford it, especially when as a consequence other big corporations get motivated by the new branding to buy more widgets of the new CEO's company.

The only pity is that often this means that entire forests are wasted when the company has to reprint all its marketing collaterals. Around 2000, my employer at the time had a big warehouse in Campbell with product brochures, but fortunately our manager had been able to convince the company to deploy a document management system and print the marketing collaterals on demand, just when they were needed.

The hard problem came when a year later the CEO decided to change the color palette. Although all brochure chunks were stored digitally, when a brochure was produced by combining chunks with the old palette and chunks with the new palette, the resulting brochure looked inconsistent.

Colleagues Hui Chao and Xiaofan Lin quickly wrote code that could perform a wide range of graphical changes to the collaterals in the repository, and this writer wrote a few lines of code that would replace an old palette color with the perceptually nearest color in the new palette. Unfortunately, already the first test run demonstrated that this was a hack that did not work in practice. For example, many chicklets ended up having bright green text on orange background, something chromatically challenged people with color vision deficiencies could not read.

The solution that worked was to use a model to compute the names of the foreground and background colors, then change one of them to the nearest color in the new palette that was at least to color name categories away from the other color. This solution ended up being very good in practice and we wrote very efficient code that could process a large repository in a very short time.

I guess a sign of good engineering is to have the intuition for an unexpected solution before the scientists have worked out all the facts, …and we did not need nuclear bomb explosions.

America Invents Act

I wrote several times, that inventions rarely come from a stroke of genius. We are pack animals, and science is a common understanding of the physical world around us, not the physical world itself. Researchers work hard to further our understanding of the world and to uncover new synergies, creating sort of an ether of knowledge. This also includes tools and technologies to further this knowledge.

From time to time, in this ether local areas condensate and great insights are born. When considered in retrospect, these insights look like strokes of genius, but they would not have been possible without a maturity attained in a certain context, or the availability of new tools.

Because of this, I like to say that ideas and inventions reach a point in time when they are ripe, and the first to recognize them is the inventor. This perspective instills a sense of urgency, because if you dither with your invention, somebody else will beat you.

In the old era of big science and large industrial laboratories, only an elite had sufficient exposure to this ether to make groundbreaking inventions. Through conferences and journals it was more or less known who works on what, and it was possible to time the filing of patents, for example so that an application is laid open the day the Tokyo Data Show opens, where a new product is unveiled.

Today, when due to the disappearance of advanced research there is less differentiation from vendor to vendor, product introductions are rarely timed to industrial fairs, but released as soon as they are ready. In computer technology for example, there are no longer a few thousand R&D personnel, but hundreds of thousands if not millions; Apple alone claims to have created 210,000 iOS app economy jobs in the US.

This huge number of technologists, combined with the increased litigation for patent infringements and the proliferation of non-practicing entities (NPE) and other patent trolls, impinges an even greater sense of urgency on protecting one's intellectual assets as fast as possible. National patent agencies support this urgency through the introduction of new processes for accelerated protection mechanisms.

Patents are not necessarily the only form of protection for one's research investment. In Switzerland, the Swiss National Science Foundation encourages patent protection mostly for inventions that are either very fundamental, or are easy to replicate. For all other inventions the recommendation is to save the money that would be spent in patent application and maintenance fees and spend it instead on the salaries for additional researchers. The reasoning is that an agile well-managed enterprise can generate new products faster than the competition can copy them, hence a defensive publication is more efficient than a patent.

In the US, the law protected the first to invent, not the first to file, so in general companies were sometimes a little slower than Swiss companies. However, this has recently changed. This picture taken September 16, 2011 at Thomas Jefferson High School for Science and Technology in Alexandria, Virginia, shows President Obama signing the America Invents Act, historic patent reform legislation that is helping American entrepreneurs and businesses bring their inventions to market sooner.

The America Invents Act helps businesses, inventors, and entrepreneurs in five immediate ways:

A fast track option for Patent Processing within 12 Months: Instead of an average wait time of almost three years, the Patent and Trademark Office will be able to offer startups and growing companies an opportunity to have important patents reviewed in one-third the time – with a new fast track option that has a guaranteed 12-month turnaround. Patent ownership is a critical factor venture capital companies consider when investing in entrepreneurs hoping to grow their business.

Reducing the current patent backlog: Under the Obama Administration, the patent backlog has already been reduced from over 750,000 patent applications to 680,000, despite a 4% increase in filings. The additional resources provided in the law will allow the Patent and Trademark Office to continue to combat the backlog of nearly 700,000 patent applications and will significantly reduce wait times.

Reducing litigation: The Patent and Trademark Office will offer entrepreneurs new ways to avoid litigation regarding patent validity, at costs significantly less expensive than going to court.

Increasing patent quality: The Patent and Trademark Office has re-engineered its quality management processes to increase the quality of the examinations and has issued guidelines that clarify and tighten its standards for the issuance of patents. The legislation gives the USPTO additional tools and resources to further improve patent quality, and allows patent challenges to be resolved in-house through expedited post-grant processes.

Increasing the ability of American Inventors to protect their IP abroad: The new law harmonizes the American patent process with the rest of the world to make it more efficient and predictable, and make it easier for entrepreneurs to simultaneously market products in the U.S. and for exporting abroad. The Patent and Trademark Office has also expanded work-sharing with other patent offices around the world to increase efficiency and speed patent processing for applicants seeking protection in multiple jurisdictions.

Peer To Patent 2011 Pilot

Peer To Patent is a historic initiative by the United States Patent and Trademark Office (USPTO) that opens the patent examination process to public participation for the first time. Peer to Patent is an online system that aims to improve the quality of issued patents by enabling the public to supply the USPTO with information relevant to assessing the claims of pending patent applications.

The Peer To Patent 2011 Pilot has opened and will run through September 30, 2011. Eleven applications have been posted for review, all in subject matter classes covering software. Patent classes covered by these apps include 380 (cryptography), 701 (vehicles, navigation, and relative location), 706 (artificial intelligence), 707 (database and file management or data structures), 709 (multicomputer data transferring), 712 (processing architecture and instruction processing), 715 (presentation processing of document, operator interface processing and screen saver display processing), and 718 (virtual machine task or process management or task management/control).

Evolution of the Silicon Valley

When I arrived in the Silicon Valley, it was the center of the universe for anything connected to silicon. Engineers, scientists, and researchers at the many think tanks were outbraining the Russians and inventing so many new technologies that still today young entrepreneurs are trolling expired patents and old papers to create new products. Some think tanks like Fairchild, SRI, and Xerox PARC generated cascading start-ups, because invention was much faster than business creation.

Parts of the Valley, like Sunnyvale, were mostly orchards. For example, when Dave Packard had a new crop on his orchard on Page Mill Road, his secretary Margaret would put a brown bag full of fruit for sale at the coffee station where engineers picked up the free doughnuts every morning.

Successful entrepreneurs had expansive houses in Atherton, Woodside, or Los Altos Hills. Palo Alto was a quaint city with a Woolworth on University Avenue and mostly inhabitated by technologists and executives at high tech companies. Houses were typically 1500 to 2000 square foot Eichlers or squarish ranch houses, with a little Accord in the driveway, and costing almost a decade's worth of salary.

Today the technologists are largely gone, replaced by investment bankers and lawyers. Technology is more lucrative as the object of patent litigation, often by nonpracticing entities, as elucidated in a recent PriceWaterhouseCoopers study.

Fortunately for executives, land in Palo Alto has become much more affordable, just a few months worth of their compensation. Therefore, contemporary needs of entertainment space for wine cellars, movie theaters, media rooms, gyms, nanny suites, etc. can be met by building 4 story houses (two underground and two above ground), because the house to parcel ratio requirement can easily be met by buying two or more adjacent lots and consolidating them.

Now that the technologists are gone, what is happening to Photonics West, one of the largest conferences in the Silicon Valley with over 15,000 attendees? Well, EI has been priced out of the Valley and will be held at the Hyatt Regency San Francisco Airport Hotel next year.

Photonics West already was in San Francisco this year. As EI came to a close, we were wondering what impact it would make to the local hotels, with this January event leaving San Jose after 20 years. I was walking with Mary and John McCann towards the Fairmont Hotel, where we were to discuss with Alessandro Rizzi what we learned from the Dark Side of Color session. We figured this former noisy Photonics West headquarters must be very quiet now.

Wrong we were! The place was full of furry animals.

Eventually, there is no impact to the local economy for us techies leaving. Photonics West has been replaced by FurCon as the annual conference in the third week of January. In the picture above: Alessandro Rizzi, Carinna Parraman and Mary McCann with a Furry in the Fairmont lobby.

For the report on the local news, click here. Food for thought.

g2 camera calibration for RIPs

There has been quite a bit of speculation about the motivation behind our work on the g⁽²⁾ camera; we have even been slashdotted last January 25^th, so we might as well open the kimono on it, at least a little tiny bit.

In fact we are now well protected after getting the necessary patents. Every year, 15 days before Tax Day, viz. on April 1^st, the Patent Office allows inventors to demo their apparatus directly to the examiners, instead of filing a written patent application. Doing a demo is very efficient. We set up shop in the enormous hall of the Madison Building (see picture below, the tables outdoors are for inventors demonstrating a perpetuum mobile). We had two rows of tables with all the prototypes, gizmos, and gadgets we had developed and demonstrated them step by step and claim by claim to the attentive Examiners.

Of course the examiners had many questions and doubts, but fortunately at the left of the hall shown in the above picture there is one of the best libraries on this planet, so while we engineers were haggling with the Examiners, our managers and patent attorneys were busy building stronger cases and strengthening the claims.

Further down on Duke Street, on the block after the Whole Foods Market, there is a Marriott Residence Inn, where we could stay six people in each suite and get a lot of quality time to polish our inventions, while our interns prepared hearty meals for us. Our patent attorneys were comfortably lodging at the Westin across the street from the Federal Court building.

But, we are digressing — back to the motivation for the g⁽²⁾ camera. In high-speed digital printing, the bottleneck has always been in the ripping (RIP, Raster Image Processor) or, in HP parlance, the DFE (Digital Front End). For example, when we were working on the Xenith system at PARC in the mid-Eighties, Nick Sheridon was running the print engine at 300 ppm using Tibor Fisli's quad-spot laser diodes, while Gary Starkweather cranked the resolution up to 4000 dpi. Yet, even after adopting the Dragon's MBus, the shipping Docutech product could only run at 100 ppm and 600 dpi due to RIP limitations.

Concomitantly, at Canon the A-printer had been developed. This poster printer had a 40 inch wide array and was printing on paper rolls using bubble jet. The head actually consisted of four 10 inch heads mounted in a staggered pattern, and it was incredibly fast. Only, it required a MasPar mini-supercomputer to deliver the bits. The galleries and poster shops in Roppongi never ordered enough printers to make the product commercially viable.

Today, our valued customers buying high-speed digital presses still have to dive deep into their pockets to buy a costly DFE. This is why in Director Gary Dispoto's Print Production Automation Lab, Dr. I-Jong Lin manages the RIP project. This is also why Dr. Ray Beausoleil — who just became an HP Fellow — moved to the Quantum Science Research Department. We need quantum information technology (QIT) to deliver affordable DFEs for our high-speed digital presses.

One of the challenges in QIT is to store quantum bits (qbits) while avoiding a collapse of the wave function or decoherence. Typically, qbits are stored in a semiconductor (gallium arsenide, GaAs) microcavity, therefore, we have to study the interface interactions of emanating photons.

More formally, we need to study the Bose-Einstein condensation (BEC) phase transition in a polariton system in a semiconductor microcavity. The macroscopic quantum degeneracy is typically detected by probing the statistical properties of light emitted from a microcavity, under the presumption that the statistics of the exciton polaritons are faithfully transferred to the emanating photons.

The figure below shows Interference fringes (a) at 770 nm wavelength used to verify the BEC of polaritons in GaAs microcavity and (b) at 546 nm measured for the green line of a pulsed Hg-Ar discharge lamp.

This figure shows, that a coherent light source (e.g., a photon laser or decaying polariton BEC) can exhibit the same first-order correlations as a chaotic (or thermal) light source (e.g. Hg-Ar discharge lamp in (b)). The table below shows that proper disambiguation of a coherent state also requires measurement of the second-order correlation function g⁽²⁾ associated with intensity noise correlations:

Function	Incoherent	Coherent	Chaotic
g(1)	0	1	1
g(2)	1	1	2

In summary, as we wrote in our slashdotted paper, the application of the g⁽²⁾ camera is to take pictures to confirm the presence of true Bose–Einstein condensates (BEC). The next challenge is calibrating the camera. As we wrote in or popular technical report on Spectrophotometer Calibration and Certification, tight calibration is very critical in this kind of applications.

Fortunately, there is a condensate that is readily available and which is produced industrially at very tight tolerances: condensed milk. Alas, in our experiments we found a caveat. From the figure below we know that we are dealing with photon pairs. To correctly calibrate the SPADs, we have to be certain that both calibration photons have exactly the same color.

As our esteemed colleague and co-blogger Steve Simske keeps warning us about, the caveat is in the rampant counterfeiting happening in the supply chain. What happens if one photon comes from the condensed milk but the other photon comes from melamine?

In our lab we have built a special spectroradiometer, which we use as a reference for the calibration instrument. We have used it to measure the spectrum of pure condensed milk and that of melamine. Here are the plots:

Now we just use the CIE formula with the color matching functions for the 2º observer:

and get the RGB values of the two photons.

Color science is about observers, and in the end what counts is whether an observer can tell apart photons with these two RGB values. The old way of doing this was to compute the ∆E*_ab value in a perceptually uniform color space or in a CIECAM02 color appearance space based on the JND (just noticeable difference).

In our work on the color thesaurus we have established that a more reliable method is to determine whether the names of the two photon's colors are synonyms. Since the first to take a picture of a BEC will almost certainly get the Nobel Prize, we decided to use the data from our Swedish corpus of the färgbenämningsexperiment.

The RGB values calculated above yield mjölkaktig vit for the condensate and snövit for the melamine. Clearly there is no match and the g⁽²⁾ camera would be calibrated incorrectly if the condensate is counterfeited. How did we solve this problem? The solution is in this Feinman diagram:

Since photons are massless bosons, time is symmetric and at an event E we can say that a first photon (signal s) comes from a second photon (idler i) when the two photons are entangled to form a biphoton, In other words, each biphoton can be regarded as forming a loop between source and detector (CC is the coincidence counter).

Entanglement is just a fancy technical term to say that the two photon share the same wave function, also known as Schrödinger equation. Since the color of a photon is given by its wavelength, by entangling two photons we make them of the same color.

In other words, all we have to do, is to entangle the condensed milk photons with the melamine photons and we can always calibrate correctly the g⁽²⁾ camera, regardless of possible counterfeiting with melamine, because the photons get the same color.

This movie shows principal color scientist Nathan Moroney in our lab entangling the photons.

Of course, the stirring requires a lot of training, because the wave function can easily collapse, so do not try this at home!

For the viewpoint of our performance analyst, see his post in the Performance Agora.

For now, having tattooed on our tonsils to finish the new RIP, we are focusing on that. Once we have delivered product, our future research ideas include feeding the entangled condensed milk photons to Schrödinger's cat and take its pictures with the g⁽²⁾ camera. We will post our images here, so stay tuned!

In the meantime, we wish you a happy April Fool's Day.

Patents versus markets

The latest printed issue of Science has a surprising article on the promotion of intellectual discovery. The link is Science 6 March 2009: Vol. 323. no. 5919, pp. 1335 - 1339. The Science editors considered this a significant report, giving is a Perspective article. The researchers found that their "markets system" performed better than the patent system.

I used the qualifier "surprising" because this is a strange time to advocate free markets instead of regulation, a time when the G20 members this weekend were in London to discuss increased regulation of the financial markets. I think the key in this argument is in the concept of the transitioning organization and its attractiveness for psychopaths as Hare and Babiak explain in they book Snakes in suits, written in the aftermath of Enron. In essence, you cannot assume all players are honest and have a conscience — you need a mechanism to cater for dishonest players, and regulation & oversight can be an effective mechanism.

The perspective article in Science list some caveats from the economist's point of view. Let me add some caveats from an inventor's point of view.

As we are now learning in the current global crisis, optimizing for short-term gains is not a robust strategy. Instead, the optimization should be for long-term success. In their paper, Meloso et al. run the experiment for a few minutes and then close the market for the day to tally the gains and losses.

Innovation runs on a very long time scale. In my experience, valuable inventions in a research lab can take 10 or more years to become commercially relevant for their assignee. In fact, technology and society have first to evolve to a point where the decision makers can grasp the significance of an invention. Then engineering has to learn the new technology and develop products. Marketing and sales staff has to be trained, along with customer support. Only then, the new technology can be advertised and brought to market with a chance of recuperating the investment and generate profits, hopefully to invest in new inventions and not in personal jets.

Patent applications are often filed to generate a big portfolio that can be used in cross-licencing negotiations, so quantity is often more important than quality. This means that the original managers deciding on an application can incorrectly guess which aspects of an invention have long-term value and should be protected, thus instructing the patent attorneys to make the strategically incorrect claims, reducing its commercial value. The proposed market model cannot account for this.

If an invention is important for the long-term technology strategy of a company, the company must keep working on it for the required decade or so, continuously filing patent applications as it refines the invention, so it will have a strong patent portfolio.

Fresh Wind in DC

A fresh wind blows in D.C. — there has been a multi-level reset; good excuse for a multi-level post, which I have not done for a while.

Across the Potomac river from the White House palace lies the town of Alexandria. As the taxi rolls off the Beltway, the driver notes the many police patrol cars. He explains a couple of days ago three kids shot and killed a taxi driver right here. All they got was a hundred dollars cash and a GPS. He continues, from the economic meltdown and all the money flowing into bailouts, there is not more a bad or a good part of town, crime is way up all over town.

We are in the part of town between Duke Street and Eisenhower Avenue, formerly called Spring Garden Farm. It is located just beyond the town old Corporation limits — and was of course exempt from taxation — along a major commercial artery, Little River Turnpike. It was the site of the Duke Street Tanyard.

Peter Wise, a city councilman and tanner, established the Duke Street Tanyard by 1797. The business was situated near a stone bridge on the east bank of Hooff's Run by West End Village. The tannery's ownership and name changed many times: Quakers operated the large tannery from 1812 until it was destroyed by fire in 1853.

West End, Alexandria's first suburb, was a processing center for cattle, which were brought here for slaughter and butchering. The hides were then taken to the tannery to be processed into leather by soaking them in solutions of lime, tree bark and animal dung. The tanned hides were curried by oiling, scrapping and pounding and then made into saddles, harnesses and boots.

Today, this area is very different and has been renamed to Carlyle District. Of course some very bad things like slaughter and butchering still happen here today, as in the building below. In the Albert V. Bryan United States Courthouse the government deals with the worst of its citizens.

However, the mindful visitor notes the Carlyle District is the habitat for a very different human being than the West End was. People walk in and out from the many huge office buildings at every hour of the night or day, seven days a week. They have the fast and decisive gait of busy professionals, scientists and engineers.

All around are fancy condominium towers, and the Whole Foods Market is an order of magnitude larger than the one in posh Palo Alto, which was their first store outside Texas and filled the entire former Oldsmobile dealership. The deli food section is particularly large and multi-cultural, a sure sign for a very busy customer base.

Are we looking at the main Microsoft or Google campus? No, those are on the West Coast.

Let us check out the monument in front of the main building. The geodesic dome must have a plaque with the sponsoring company name.

Oh!, everything in the Carlyle District is labeled with an inventor name and a patent number. Well, almost everything; some establishments may have an actual name, like the bar and grille for the lawyers doing business in the courthouse:

Yes Virginia, the Carlyle district is the home of the United States Patent And Trademark Office

and most buildings carry the sign below, even when they are not light blue in the map above. It seems a new building goes up every year.

So, what is this fresh wind? The USPTO in Alexandria has nothing to do with the old USPTO that was across the Potomac river in D.C. It is not a bureaucratic entity, but an engine of growth provided by the Department of Commerce for the benefit of society.

The USPTO now offers many services for those who want to contribute to society through technology. Check out their web site at http://www.uspto.gov/ and you will be pleasantly surprised at the many available services to make you and your technology business successful.

As you see right up on top at the right side of their home page, the USPTO is hiring. It is maybe the last organization in the U.S. hiring technologists in these troubled times.

Today's examiners are skilled scientists and engineers. They work in modern closed offices with the latest and best tools; gone are the shoes from yesteryear. They research each application with the same competence, skill, and care as the technologist who wrote the application.

Despite all the new services, the main function of the Patent Office is to protect the inventors of a technology so they can recuperate their research investment. This is especially important today.

The lesson learned from the lost decade in Japan after their bubble economy burst in 1993, is that the way a society digs itself out of a deep economic crisis is that companies start investing in new manufacturing equipment so they can produce products superior to those of the competition. It is a slow spiral and it is powered by technology.

Therefore, hurry up and make a breakthrough invention, then rush to Alexandria

and file a patent application.

The lesson from Japan is why it is good to see that the USPTO is still actively hiring.

However, when you stroll the streets of the Carlyle district or choose your meal at the Whole Foods Market's vast deli, you hear troubling things. The budget is very tight, and the managers are struggling. Some training is deferred and the support personnel is scarce. There is a sense of urgency and apprehension in the air.

The government needs its money to bail out the industry. There is a fear that the snakes in suits have come up with a new milking scheme. Instead of grabbing the consumers by the ankles and shaking out the money in their pockets, they now go to the government for a bailout and let the government take the taxpayers by the ankles and shake out their money.

If this becomes the new modus operandi, then we are doomed. If you encounter a snake in suit take action. These entrepreneurial pretenders are only a small minority of the executives, the other 96.5% are genuine. Make sure they put their technologists on the spiral out of this economic crisis.

A stop sign does not mean that you should fall asleep on the wheel. After you stopped, looked for crossing traffic and pedestrians, and checked your destination, press on. Urgently, please…