Planet R

Sameer Chopra, vice president of Advanced Analytics at Orbitz Worldwide, wrote recently in Analytics magazine about the changing landscape of processes, software and systems for statistical modelers. In a section on "Big Data and Open Source Analytics", Chopra lays out the reasons why the R language "has become the data-mining tool of choice for machine learners":

• R has very good integration with Hadoop, an area where established commercial statistical tools have frankly been playing catch-up over the past year. (Note: At the time of this writing, some established statistical solution providers were announcing an access interface to Hadoop.)
• Many startups and smaller firms do not have deep pockets and are embracing open source tools such as the R programming language and NoSQL database systems such as MongoDB.
• R is a leading language for developing new statistical methods, and it is a platform for statistical innovation and collaboration across both the corporate world and academia. In my opinion, for the first time in years, the stronghold of established commercial players seems to be potentially threatened; open source tools are better suited for Big Data and will slowly but surely continue to take share away from commercialized statistical packages. In fact, traditional statistical vendors have recognized that R is a force to be reckoned with. In response, many of these vendors have developed hooks into R so users can interface with the R language.
• Based on the resumes I’ve been reading, the next generation of data miners is flocking to R as their go-to tool. Professors in general are comfortable with R; they tend to use R and Excel as part of their curriculum.
• In short, open-source analytics tools and platforms have arrived.

Chopra says that the usage of R in the commercial sector is growing "as firms such as Revolution Analytics focus on the enterprise capabilities for R" (for example, Revolution R Enterprise's Hadoop support and enterprise deployment).

Chopra also has some interesting perspectives on statistical modeling vs machine learning which you can find in the full article linked below.

Analytics magazine: The times they are a changin’ for advanced analytics

R user Arthur Charpentier attempts to use the raster library and R functions to find Waldo in a "Where's Waldo" image:

Sadly, it turned out that Waldo was a bit too tricky to spot using these techniques. But Arthur did have more success identifying the US flag in a shot from the Apollo mission, and identifying answers in the form for a multiple-choice test. All of the R code is provided at the link below, so that's a great place to start if you're looking to do some image analysis in R yourself.

Freakonometrics: Finding Waldo, a flag on the moon and multiple choice tests, with R

The most recent edition of the Revolution Newsletter is out. The news section is below, and you can read the full May edition (with highlights from this blog and community events) online. You can subscribe to the Revolution Newsletter to get it monthly via email.

New R Training Courses Announced. Three new R courses from leading R experts are now available for registration:

• An Introduction to R for SAS, SPSS, and Stata Users will be presented by Bob Muenchen (author of R for SAS and SPSS users) June 26-29. This is an on-line workshop with live instruction that you can attend from your desk - no travel necessary!
• Visualization in R with ggplot2 is also a new web-based course with live instruction from Garrett Grolemund & Dr. Winston Chang of Rice University, presented online June 19-20.
• R Development Master Class will be presented in-person by R package author and professor Hadley Wickham, June 21-22 in NYC and June 28-29 in Redwood City, CA.

Sign up to these courses now to secure your seat -- attendance is limited!

On the hunt for a cure for MS. Researchers at SUNY Buffalo are using big-data analytics and generic data toresearch a cure for Multiple Sclerosis. Read in Forbes how Revolution R and IBM Netezza speeds up their research, or join the #IBMDataChat discussion on Twitter at noon ET on Thursday, May 10 to learn more.

More free R webinars. Revolution Analytics' Spring webinar series continues, with upcoming presentations on Getting Up to Speed with R,  spatial statistics with R, data mining with R, Achieving High-Performing, Simulation-Based Operational Risk Measurement with RevoScaleR, and a look at the next version of Revolution R Enterprise.

Catch our archived webinar replays. Missed any of the recent presentations on Big Data with R and Hadoop, integrating R with BI tools and MS Office, or How Big Data is Changing Retail Marketing Analytics? Check our webinar archives for streaming replays and presentation downloads. 

Putting the R in Analytics. A feature article in Information Age reveals how new graduates are driving adoption of R in industry.

Upcoming Conferences for R Users. Revolution Analytics is proud to sponsor: R/Finance 2012 (the premier conference for R users in the finance industry) on May 11-12 in Chicago; and the Interface 2012 conference on the future of Statistical Computing, May 16-18 in Houston.

R user conference deadlines approaching. If you're planning to attend useR! 2012 in Nashville but want to avoid late registration fees, the deadline for regular registration is May 12. 

Revolution Analytics: Newsletter Archive

Based on an analysis of Google Scholar data on usage of statistical software, Bob Muenchen makes a forecast: R will overtake SAS and SPSS in 2015. Forecasting is extrapolation — always a tricky business — so Bob also provides these qualitative reasons why R will continue to grow at the expense of SAS and SPSS:

• The continued rapid growth in add-on packages (Figure 10)
• The attraction of R’s powerful language
• The near monopoly R has on the latest analytic methods
• Its free price
• The freedom to teach with real-world examples from outside organizations, which is forbidden to academics by SAS and SPSS licenses (it benefits those organizations, so the vendors say they should have their own software license).

See how Bob comes up with this forecast (using R, of course!) at the link below.

r4stats.com: Will 2015 be the Beginning of the End for SAS and SPSS?

We had a great Twitter conversation last Thursday on the use of big-data analytics, Revolution R Enterprise, and IBM Netezza in the search for a cure for MS. Many thanks to the other panelists: Murali Ramanathan (SUNY Buffalo), Tim Coetzee (National MS Society) and moderator Shawn Dolley (IBM) for fielding and answering questions from interested parties following #IBMDataChat. As you can see from this twitteR analysis, it was a lively discussion, with more than 300 tweets during the designated hour:

IBM's James Kobielus has a summary of the chat, highlighting some of the key nuggets of information. For example, Dr Ramanathan revealed that this research is helping to understand interactions between genetic factors and environment that could help identify lifestyle and diet changes to manage MS. 

Also, in this interview with IBM's Mike Kearney I explain why the R language is uniquely suited for research like this, and how Revolution R works with the massive data volumes required for this study.

Thinking Inside the Box: Are you ready for big data? R is.

Looking to learn R, or to expand your R skills for data visualization or package development? Here are some R courses presented by the experts you may be interested in:

June 19-20: Visualization in R with ggplot2. This course presented by Garrett Grolemund & Dr. Winston Chang of Rice University is also a web-based course with live presentation. This course provides instruction on data visualization with R, including data transformation, visualization of Big Data and polishing graphics for presentation.  

June 21-22 (in New York City) and June 28-29 (in Redwood City, CA): R Development Master Class is an in-person, in-depth R course presented R package author and professor Hadley Wickham. This two-day course offers expert instruction in R programming and package development, and is ideal for anyone looking to hone their R development skills with expert instruction.

June 26-29: An Introduction to R for SAS, SPSS, and Stata Users. This course will be presented by Bob Muenchen, author of R for SAS and SPSS users and R for Stata Users. This course will let you build on your existing statistical software skills to learn the R language -- all without leaving your desk. The course features live presentation and student/teacher interaction via a web-based course delivery system.

For more information about these courses (including pricing and registration), please follow the link below.

Revolution Analytics: Public Training Courses

When I was a kid growing up in Australia, it seemed like every commercial break during the Saturday morning cartoon's or after-school shows was punctuated by some PSA encouraging us to lead a healthier life. These "community service announcements" were government-sponsored, and often paired a low-budget animations with a catchy jingle. Strangely enough, lots of Australians (me included) remember them fondly, and can still recite the songs on demand. Here are a few of my favourites:

"Slip Slop Slap" made avoiding skin cancer fun (an important lesson in the Sunburnt Country):

There were a whole series of these "Life. Be in it." ads, encouraging us all to get out of the house and do something lest we end up like Norm. (Seems like these ads are undergoing a revival.) I'm pretty sure there was one in the series encouraging Australians to have "one alcohol free day a week", but I'm either misremembering or it's not on YouTube.

But by a long shot, my absolute favourite was the Vitamins Song:

I always hoped there'd be a sequel where the Slip Slop Slap Magpie and Vitamin D resolved their differences in a Battle Royale, but sadly it never came to be. Enjoy your weekend!

Kevin Quealy, graphics editor at the New York Times, has published another fascinating behind-the-scenes look at how the Times creates data visualizations for print and online. In his latest post, he looks at how a visualization of the Yankee's Mariano Rivera performance compared to other Major League Baseball pitchers was created. (Detail below, click for the full image.) 

The infographic began its life as a hand-drawn sketch, that begat a line-chart created using R (based on data scraped from the Web). The R chart was was then cleaned up and annotated using Adobe Illustrator for publication. One interesting detail of the process: the source R graph is deliberately created using garish colours (purples, greens, etc.) to make the color-selection process easier in Illustrator.

Check out the ChartsNThings archive for other great studies of the process of data journalism. Many of the case studies involve the use of R code, such as these visualizations of visitors to the White House, Santorum's primary support, Santorum/Romney exit poll data, NFL players mentioned on ESPN, the defense budget and the richest 1%.

ChartsNThings: Sketches: How Mariano Rivera Compares to Baseball’s Best Closers

Turns out that Megan would have never gotten a callback for an audition. (Via Ben Schmidt.)

In case you missed them, here are some articles from April of particular interest to R users.

Information Age published a feature article on R, describing how new graduates are driving adoption of R in industry.

Bob Muenchen has updated his list of R package equivalents to SAS and SPSS procedures.

A history of Data Science, including Bill Cleveland's 2001 paper.

Researchers at SUNY Buffalo are using Revolution R and IBM Netezza with genetic data to research a cure for Multiple Sclerosis, and the story has been reported in Forbes, eWeek and other media.

Pairach Piboonrungroj has compiled a list of 20 free R tutorials from around the world.

The annual Rmetrics financial engineering workshop takes place in Switzerland, June 24-28.

An elegant solution to a pairs-of-squares sequence puzzle, based on graph theory.

An example of using R to build a recommendation engine, and ranking the most popular movies from the million row movie dataset.

When is Big Data useful for statistical analysis? Norman Nie provides five examples in the Sybase Capital Markets Guide.

Revolution Analytics' Spring webinar series is underway, with presentations on Big Data with R and Hadoop, integrating R with MS Office, spatial statistics with R, data mining with R and retail marketing analytics.

The US National Oceanic and Atmospheric Administration uses R to forecast river flooding events.

R continues its growth in academia (as measured by Google Scholar citations); SPSS and SAS see steep declines.

A fantastic animation of 18th-century sailing ship voyages, created with R.

R and other open source tools used at the Consumer Financial Protection Bureau.

An introduction to the new Julia language, and a comparison with R.

SAP's HANA in-memory datastore provides integration with R.

Saraj Gupta has written an in-depth article on how the internals of R's name lookup mechanism works.

LityxIQ uses R functions glm, MASS, rpart, nnet and rjson for their online marketing analytics and optimization application.

Google can graph 2-variable and 3-variable equations.

Other non-R-related stories in the past month included: an animation of world ocean currents, how English sounds to Italians, creating the Pharoah's Serpent effect with mercury thiocynate, and a unique performance of "Somebody that I Used to Know".

There are new R user groups in Milan and Cologne. Meeting times for local R user groups can be found on the updated R Community Calendar.

As always, thanks for the comments and please send any suggestions to me at david@revolutionanalytics.com. Don't forget you can follow the blog using an RSS reader like Google Reader, or by following me on Twitter (I'm @revodavid). You can find roundups of previous months here.

In a blow to SAS's efforts to litigate competitor and low-cost SAS clone WPS out of existence, the European Union High Court has ruled that programming languages can't be copyrighted. SAS Institute (Cary, NC) had claimed that the WPS software — which allows users to process SAS data files and SAS "data step" scripts without SAS software — breached SAS's copyright in its re-implementation of SAS functionality. But given that WPS did not study SAS source code, and merely reimplemented its interfaces and behaviour based on observation, the court ruled that this was not a violation. In a press release following the decision, the court stated:

 ... neither the functionality of a computer program nor the programming language and the format of data files used in a computer program in order to exploit certain of its functions constitute a form of expression. Accordingly, they do not enjoy copyright protection.  

(Emphasis in press release.) SAS has issued no comment on the ruling to date.

The case has implications beyond SAS and WPS. In Europe, at least, it implies that developers of both proprietary and open-source software have the right to duplicate the functionality and exposed interfaces (in this case, the structure of SAS procedure calls) of a proprietary language provided they don't copy the source code of the implementation itself.

The EU ruling conflicts with that of the jury in the Google v Oracle trial, where it was decided that Google's re-implementation of Java APIs in the Android OS did infringe Oracle's copyright. Implementing a SAS procedure and a Java API are similar programming efforts, and the EFF has argued convincingly that preventing programmers from re-implementing APIs harms innovation. Nonetheless, the US-based jury did not decide if Google's Android implementation was a fair use infringement of copyright; if that question were to be resolved in Google's failure the outcome would have a similar practical effect as the EU decision.

JDSupra: European Union Court Rules that Software Functions Cannot Be Copyrighted 

In yesterday's webinar, Revolution Analytics CTO David Champagne demonstrated how to integrate statistical graphics and analytic computations created using R software with a variety of third-party applications. In each case Revolution R Enterprise Server is running as a compute server to the client application, with R scripts launched on each user interaction via the RevoDeployR Web Services API. David demonstrated five examples of such integration (watch the demo by clicking on the links below, and switch to full-screen for easier viewing).

David showed three different ways of delivering interactive sales forecasts in client applications:

• An HTML 5 application that offers sales forecasts based on a user-selected history from product sales data. This type of interface would be suited to mobile devices like an Android smartphone or iPad. 

• An enhanced QlikView report, including a sales forecast from patio furniture sales data, with options for the type of forecast model and with data visualization from both R and QlikView.

• An enhanced Jaspersoft BI Dashboard, with an interactive sales forecast based on T-shirt sales.

David also showed two examples where the end-user has even more control over the parameters of the analytic computation:

• A custom interactive web-application written in JavaScript, to perform market basket analysis on retail transaction data. The distribution of purchases is rendered on an interactive map, and products commonly purchased together (within a selected sub-region) are displayed as a tree chart and as association rules. 

• A Microsoft Excel spreadsheet, with a custom toolbar button that displays a regression dialog, with the regression results (from R) embedded directly in the same spreadsheet the source data was taken from.

The great thing about integrating R into client applications in this way is that:

• The R computation is run on-demand, and with the results presented in context to the end user;
• The end user doesn't need to know R (in fact, they're probably don't even know R is involved), while still providing some control over the computation done in R.
• R doesn't need to be installed on the client device. In fact, Revolution R Enterprise will most likely be running on a dedicated remote server, in a data center or even in the cloud.

You can see David's entire presentation (including details about the client/server architecture and the Web Services API that makes all of this possible) in the webinar replay below, or by downloading the webinar slides.

Revolution Analytics webinars: Calling All Data Scientists and Web Developers! Integrate Your Advanced Analytics into BI Apps and MS Office and Multiply Their Value