Friday, April 27, 2018

Data Analysis Careers

On 25 April 2018, the European Commission increased its investment in AI research to €1.5 billion for the period 2018-2020 under the Horizon 2020 research and innovation program. This investment is expected to trigger an additional €2.5 billion of funding from existing public-private partnerships, for example on big data and robotics. It will support the development of AI in key sectors, from transport to health; it will connect and strengthen AI research centers across Europe, and encourage testing and experimentation. The Commission will also support the development of an "AI-on-demand platform" that will provide access to relevant AI resources in the EU for all users.

Additionally, the European Fund for Strategic Investments will be mobilized to provide companies and start-ups with additional support to invest in AI. With the European Fund for Strategic Investments, the aim is to mobilize more than €500 million in total investments by 2020 across a range of key sectors.

With the dawn of artificial intelligence, many jobs will be created, but others will disappear and most will be transformed. This is why the Commission is encouraging Member States to modernize their education and training systems and support labour market transitions, building on the European Pillar of Social Rights.

The annus mirabilis of deep learning was 2012 when Google was able to coax millions of users into crowdsourcing labeled images. They also had tens of thousands of servers that were not very busy at night. Most of all, however, Google has an incredible PR department that was able to create a meme.

  1. Software defined storage (SDS) on commodity hardware made it very inexpensive to store large amounts of data. When the cloud is used for storage, there are no capital expenditures.
  2. Ordinary citizens became willing to contribute vast amounts of data in barter for free search, email, and SNS services. They were also willing to label their data for free, creating substantial ground truth corpora that can be used as training sets.
  3. High-frequency trading created a market for GPGPU hardware, resulting in much lower prices. Also, new workstation architectures made it possible to break the impasse caused by the end of Moore's law.
  4. ML packages on CRAN made it easy to experiment with R. Torch and Weka made it easy to write applications capable of processing very large datasets.

Many companies are setting up analytics departments and are trying to hire specialists in this field. However, there is great confusion on what the new careers are and how they are different. Often, even the companies posting the job openings do not understand the differences.

Recently, in the Sunnyvale City Hall, two representatives from LinkedIn and a representative each from UCSC Silicon Valley Extension and California Science and Technology University, participated in a panel organized by NOVA, dispelling the confusion.

Essentially there are three professions: data analyst, data engineer, and data scientist:

  • Data analysts tends to be more entry level and do not necessarily need programming or domain knowledge: they visualize data, organize information and summarize data, often using SQL. Essentially, they deal with data "as is."
  • Data engineers do what is called data preparation, data wrangling, or data munging. They pull data from multiple, distributed (and often unstructured) data sources and get it ready for data scientists to interpret. They need a computer science background and should be skilled with programming, Hadoop, MapReduce, MySQL, and Spark.
  • Data scientists turn the munged data into actionable insights, after they have made sure the data is analytically rigorous and repeatable. They usually have a Ph.D. The ability to communicate is vital! They must have a core understanding of the business, be able to show why the data matters and how it can advance business goals and communicate this to business partners. They need to convince decision makers, usually at the executive level.
data analysis careers

Monday, March 26, 2018

Stanford Workshop on Medical VR and AR

5 April 2018, there will be a public workshop on medical head-mounted displays in Stanford. The workshop is designed to support collaborations between the engineers who are developing VR and AR technologies and the surgeons and clinicians who are using these technologies to treat their patients.

The workshop features talks by researchers who are developing VR and AR technologies to advance healthcare and panel discussions with Stanford physicians who are using VR and AR applications for surgical planning and navigation and for alleviating pain and anxiety in their patients.

There will be an interactive demo session featuring research projects, clinical applications, and startup ventures.

Seating is limited, so if you wish to attend, we recommend that you register now at the website

Stanford Workshop on Medical VR and AR

Monday, February 12, 2018

Claudio Oleari

On 23 January 2018, Claudio Oleari passed away at the age of 73 in Reggio Emilia. He was the last and ultimate authority on the OSA-UCS color space and perceptually uniform color.

He was an eminent physics scholar and an associate professor at the University of Parma, at the Department of Physics and Earth Sciences. He devoted his life to the activities of teaching with the same passion and interest that he dedicated to research in the context of color, applying physics to perception and establishing its role in colorimetry. In 1995 he started the Gruppo in Colorimetria e Reflectoscopia, which later became the Associazione Italiana Colore.

His availability for colleagues and students and his ability to listen and advise are proverbial: his kindness will always be remembered by everyone who has known him. These qualities are exemplified by the message on his profile at the University of Parma “You are welcome any day and at any time, even without an appointment. It is useful to verify by telephone my presence in the office. To book a meeting and ask questions, sent an email to”

He initiated, within the Italiana Association, many valuable informational activities and forged many connections which persist as a rich bibliography, always having in mind the need to invest in research and training both in Italy and abroad.” He initiated, within the Italiana Association, many valuable informational activities and forged many connections which remain as a rich bibliography, always having in mind the need to invest in research and training both in Italy and abroad.

His death leaves a void difficult to fill, and the world of color loses an intellectual and an attentive and informed scholar.

Claudio Oleari

Thursday, January 25, 2018

Perceptual Similarity Sorting Experiment

If you have an extra ~5 minutes please try out our online perceptual similarity sorting experiment.

This is follow-up to the work that Michael Ludwig (one of our summer interns from last summer) conducted and is continuing to work on as part of his PhD research.

For more details, please see this about page for the experiment. Thank you.

Friday, January 12, 2018

Annotating detected outliers

The so-called Twitter Anomaly Detection function for R is excellent but also very minimalistic. The input is a two-column data frame where the first column consists of the timestamps and the second column contains the observations. In addition to a plot, the output is a data frame comprising timestamps, values, and optionally, expected values.

In practice, we usually have some semantic information that we would also like to include in the output, so we do not have to refer back to the original data. Fortunately, there is a quick-and-dirty way to add a description to the outlier data frame.

We start with the annotated data frame containing at least columns with the timestamps, the observations, and factors providing contextual or semantic information on each observation. We then create a simple data frame with just the first two columns, which we pass to the outlier detection function.

We can write a trivial function that for each outlier finds the row index in the simple data frame and looks up the semantic information in the annotated data frame:

AddDescription <- function(series1, series2, outliers) {
 quantity <-  lengths(outliers$anoms[1])
 if (quantity < 1) return (NULL)
 else {
   result <- NULL
  for (i in 1:quantity) {
   rowIndex <- which(series1$timestamp == outliers$anoms$timestamp[i])
   newRow <- data.frame(outliers$anoms$timestamp[i],
   result <- rbind(result, newRow)
  colnames (result) <- c("timestamp", "outlier_value", "description")
  return (result)

This function is just an elementary example. It is easy to add to each outlier more detailed information you can compile from the full data frame.

Time series with outliers at green markers

outliers with descriptions
  timestamp outlier_value description
2017-01-17 06:53:00
gear display flashing
2017-09-19 09:10:00
gear shift failure
2017-11-17 07:26:00
check engine lamp on

Dates are a sore point of analytics: they alway get you. When no time zone is specified, i.e., tz = "", R assumes the local time zone. In the data frame returned by Twitter's AnomalyDetectionTs functions, the time column has UTC as the time zone. Therefore, the following statement is useful after the call to AnomalyDetectionTs:

anomalies$anoms$timestamp <- as.POSIXct(anomalies$anoms$timestamp, tz = "")

Monday, December 18, 2017

Lakota Waldorf School Fighting Poverty on an Indian Reservation

If you are still looking to give a Christmas or year's end gift that can make a big impact, consider a school in one of the poorest counties in the USA. It is the Lakota Waldorf School in Kyle, SouthDakota, the only Waldorf School on an Indian Reservation.

Swiss native Isabel Stadnick is one of the founders and current administrator, and her husband Robert Stadnick is a tribal member. In 1992, they traveled to the Goetheanum with two additional Lakota tribal members, when Dr. Heinz Zimmermann—the head of the Waldorf movement—encouraged the founders to incorporate the Lakota language and culture.

The Lakota Waldorf School's mission is to empower the children and initiate their educational process with creativity, positivity, community and Lakota culture. The Lakota Waldorf School is a small school, surrounded by never-ending prairie, in the midst of the Pine Ridge Indian Reservation. This reservation is one of the poorest counties in the United States, with an unemployment rate of 75% to 80%. Many of the local people suffer from severe alcohol and drug abuse, and much of the reservation is considered a food desert.

Because of these circumstances, the Lakota Waldorf School is an incredible support system for the 24 children who attend the school. They provide the children with wholesome meals and send them home on Friday afternoon with a weekend pack filled with healthy snacks since many of the families do not have the resources for a nutritious meal.

Each morning, the children are greeted with the wonderful smell of a healthy breakfast of oatmeal, scrambled eggs from their chicken or rice pudding with honey and raisins. Lunch consists of only organic food, vegetables from their garden and bison meat from a local store. All meals are cooked at the school.

The sixteen kindergartners and eight first and second graders that make up the Lakota Waldorf School, begin their day with the morning verse in the Lakota language, Lakota songs, music, and stories. The curriculum includes language, arts, math, science, and social studies as well as handwork, flute music, painting, drawing and modeling classes and storytelling throughout the day.

Currently, the entire school consists of one small building which houses the kindergarten, kitchen, and office. There is a separate small building for the first and second grade. To continue supporting students and their families, they are planning to add grades 3, 4 and 5 and up to 8th grade in the coming years. Plans are also underway to build an urgently needed additional building, housing a bigger kitchen, three or four additional classrooms and a healthy café shop. The new building would be built with only straw bales and solar and wind energy. Jeff Dickinson, well known as a Waldorf and a clean energy architect, is involved in the addition of the school.

Not only is Waldorf education important for these children, but the support they receive is crucial to their overall well-being. The families cannot afford to pay tuition. Therefore, the school is 100% donation-funded. The Lakota Waldorf School, the administration, current and future students and families would appreciate any donation, small or large, to sustain Waldorf education on the Pine Ridge Reservation.

People who volunteered and spent a week working with the students, who are growing up in severe poverty and some in traumatic circumstances, can personally attest to the positive impact the school has on each of their lives. The sincere hope is that the Lakota Waldorf School will continue to thrive and educate young ones for years to come.

Please visit the website and consider making a donation to ensure the survival of this extraordinary school.

Lakota Waldorf School

Monday, December 4, 2017

3D printing of bacteria into functional complex materials

A team from the ETH in Zurich and the University College in Dublin has been able to demonstrate a 3D printing approach to create bacteria-derived functional materials by combining the natural diverse metabolism of bacteria with the shape design freedom of additive manufacturing.

They have developed a biocompatible hydrogel with optimized rheological properties that allows for the immobilization of bacteria into 3D-printed architectures at a high accuracy. They have demonstrated two applications: degrading environmental toxins, and making cellulose, which can be used as scaffolds for skin replacements and coatings for biomedical devices that help protect patients against organ rejection.

Immobilization of Pseudomonas putida, a known phenol degrader, when printed allows to degrade phenol into biomass, showing the potential of the 3D bacteria printing platform for biotechnological applications. Immobilization of Acetobacter xylinum in a predesigned 3D matrix enables the in situ formation of bacterial cellulose scaffolds on nonplanar surfaces, relevant for personalized biomedical applications.

Science Advances 01 Dec 2017: Vol. 3, no. 12, eaao6804 DOI: 10.1126/sciadv.aao6804

Schematics of the 3D bacteria-printing platform for the creation of functional living materials

Tuesday, November 14, 2017

AAAS Statement on Scientific Freedom and Responsibility

Scientific freedom and scientific responsibility are essential to the advancement of human knowledge for the benefit of all. Scientific freedom is the freedom to engage in scientific inquiry, pursue and apply knowledge, and communicate openly. This freedom is inextricably linked to and must be exercised in accordance with scientific responsibility. Scientific responsibility is the duty to conduct and apply science with integrity, in the interest of humanity, in a spirit of stewardship for the environment, and with respect for human rights.

For more information:

Camille Flammarion: "Urbi et Orbi”, in L'atmosphère: météorologie populaire, 1888

Where the sky and the Earth touch

Thursday, November 9, 2017

Panasonic buying deep learning startup

Arimo was born Adatao in 2013 and is being acquired by Panasonic. It calls its product Behavioral AI and targets it to machine learning for Industry 4.0.

It started with two tools. pAnalytics is a Spark environment providing an API where developers can work with the data and expose it to the end users with charts and graphs. pInsights is the end user layer, which takes natural language queries. This tool learns from the end user's interactions and can suggest possible queries.

This approach is used to learn from the past behavior of equipment to identify complex anomalies that are hard to predict with traditional statistical modeling. The same deep learning algorithms can also be used to predict retail shopper's behavior to offer them incentives and optimize store inventories. A related solution area is financial services, where the technology can find signals and anomalies in large-scale transactional data to detect fraud, model risk, and predict investor or consumer behavior.

Panasonic first aims to apply the technology to data on business refrigerators for supermarkets and convenience stores. It envisions a service reducing energy consumption for a store chain overall by setting optimal operating patterns for individual stores, based on past data on refrigerators' internal temperature and energy use. Panasonic can then expand the application to industrial air conditioners.

In the future, Panasonic plans services to manage the physical health of the elderly based on data from appliances and a range of sensors. Since Panasonic has few data analysis experts, Arimo will be a training ground for its employees.