A Month Of Saturdays Playlist : May 2013


“Random Access Memories” joins Mylo’s “Destroy Rock & Roll” and Chromeo’s “Business Casual” in my top 3 “albums that blow you away”

Source: Spotify

BalletBoyz - “Void”

Machine Learning 2 : Reinforcement Learning vs. Supervised Learning

“Reinforcement learning (RL) and supervised learning are usually portrayed as distinct methods of learning from experience. RL methods are often applied to problems involving sequential dynamics and optimization of a scalar performance objective, with online exploration of the effects of actions. Supervised learning methods, on the other hand, are frequently used for problems involving static input-output mappings and minimization of a vector error signal, with no explicit dependence on how training examples are gathered. As discussed by Barto and Dietterich (this volume), the key feature distinguishing RL and supervised learning is whether training information from the environment serves as an evaluation signal or as an error signal…”

[Source :  Supervised Actor-Critic Reinforcement Learning - paper discussing problems where both kinds of feedback ( evaluation signal/error signals ) are available to a learning system at the same time) ]

“Having discussed key differences between reinforcement learning and supervised learning, the question arises as to whether these differences are fundamental or merely superficial differences that can be eliminated with suitable problem reformulation.”

[ Source :
Reinforcement Learning and its Relationship to Supervised Learning ]

Further reading:

Reinforcement Learning Architectures

Supervised Actor-Critic Reinforcement Learning

Generalized Model Learning for Reinforcement Learning in Factored Domains

Autonomous Helicopter flight via Reinforcement learning

Real-time reinforcement learning in continuous domains

Reinforcement learning for rule extraction from a labeled dataset

Reinforcement Learning by Comparing Immediate Reward

Maximum reward reinforcement learning: A non-cumulative reward criterion

http://www.compapp.dcu.ie/~humphrys/PhD/ch2.html

http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node7.html

http://pages.cs.wisc.edu/~finton/what-rl.html

Machine Learning 1 : Overview

These notes are my own observations and questions from my study of the material on Stanford’s Engineering Everywhere Machine Learning Course

Machine Learning :  a scientific discipline concerned with the design and development of algorithms that allow machines (computers) to make decisions or even evolve behaviors based on collection of empirical data.

The algorithms of machine learning fall into the following primary categories ( source : Wikipedia ) :

  • Unsupervised Learning :  the problem of trying to find hidden structure in unlabeled data
  • Supervised Learning : inferring a function from labeled training data
  • Reinforcement Learning : concerned with how an agent ought to take actions in an environment so as to maximize some notion of cumulative reward

( There are other classifications of algorithms, however these three classes are the major types discussed in the reference material I’ve encountered so far  )

Supervised vs. Unsupervised Learning

We can see that Supervised Learning and Unsupervised Learning are mutually exclusive categories if the training data is either all labelled or all unlabelled.

What does it mean to say data is labelled ?

Alternative definitions of the Supervised and Unsupervised learning from a Stanford University glossary help us see what it means to say data is labelled:

  • Supervised Learning:  Learning techniques used to learn the relationship between independent attributes and a designated dependent attribute (the label)
  • Unsupervised Learning : Learning techniques that group instances without a pre-specified dependent attribute.

We can say from these definitions that training data is said to be labelled when the data being analysed is comprised of pairs consisting of a set of independent input attributes and a dependent output attribute (the label), whereas with unlabeled data there is no pre-specified dependent attribute.

When a dataset is unlabeled, we may wish to use an Unsupervised Learning algorithm to find patterns within the structure of this data.

When a dataset contains labels, the label is often referred to as the “correct” or desired answer, and may wish to use a Supervised Learning algorithm to infer a function which predicts this “right answer” given a set of input attributes.

“To describe the Supervised Learning algorithm slightly more formally, our goal is, given a training set, to learn a function h : X → Y so that h(x) is a “good” predictor for the corresponding value of y. For historical reasons, this function h is called a hypothesis.”

( here X denotes the space of input values, and Y the spaces of output values.)

[Source: CS229 Lecture Notes - 1 ]

( The terms predict and predictor are highlighted in the above text as we will shortly attempt to define what we mean by these terms in more detail. )

Examples of Unsupervised/Supervised Learning

If we had a dataset consisting of the location coordinates of houses within a given city, we could say this is an unlabelled data set and we could use an Unsupervised Learning algorithm to find clusters of homes in the city.  

If our dataset included the price of each house in addition to the location attributes, we could use a Supervised Learning algorithm to look for a function which predicted the price of the house given its location.

Where does Reinforcement Learning fit in - is it Supervised or Unsupervised Learning?
 

Reinforcement Learning : concerned with how an agent ought to take actions in an environment so as to maximize some notion of cumulative reward

“Somewhere between supervised and unsupervised learning is reinforcement learning. Reinforcement learning is primarily used in control systems that might take a sequence of steps over a period of time to arrive at a desired outcome. We may know what that outcome should be, but we may not know whether any particular step is a good one (advances us toward the goal) or a bad one. Like unsupervised learning, reinforcement learning systems are trained with unlabeled data (the “right answer” is not given), but are then “rewarded” for making positive progress or achieving a goal, or “punished” for poor progress or an undesirable outcome. “Positive progress” may be counter-intuitive if it is sometimes necessary to proceed in ways that temporarily seem wrong in order to avoid an obstacle, and reinforcement learning has a general “credit assignment problem” across all the steps in a sequence when evaluating how to apportion blame for an undesirable outcome to the intermediate steps that achieved it. It would be incorrect to assume, for example, that applying the brakes caused crashes in autonomous cars simply because every failed self- driving attempt terminated with the brakes applied as the control system tried to avoid the crash. Instead, blame (negative credit) should be levied to the actions that led, perhaps much earlier, to the crash”

[ Source :  “Machine Learning, Cognition, and Big Data
by Steve Oberlin, Senior Vice President and Distinguished Engineer, CA Technologies” ]



A Month Of Saturdays Playlist : April 2013

Making Connections with Spring Social

Craig Walls talks at SpringOne 2GX 2012 about Spring Social.

Check out the “Spring Social Project Family” slide.  Feel proud that my Last.Fm, SoundCloud and Mixcloud modules are listed as members :-)

exfm-oauth-api-proxy

exfm-oauth-api-proxy : A proxy to Ex.Fm’s API supporting OAuth2 Authentication - currently deployed to exfmproxy.socialsignin.org

(source code at https://github.com/socialsignin/exfm-oauth-api-proxy )

Some time ago I wanted to provide “login with Ex.Fm” functionality on a number of my websites and to access authenticated methods of Ex.Fm’s API on behalf of my logged in users.

I started work on a spring-social-exfm module that acts as a Java client, but as Ex.Fm’s API does not currently support OAuth this module was limited in terms of it’s compatibility with the Spring Social framework, requiring a user to explicitly provide a username and password to access authenticated API methods.

I started experimenting with creating a proxy to Ex.Fm’s API that supports OAuth2 authentication, which would then mean I could develop a Spring Social module for Ex.Fm which could be used as an OAuth client should Ex.Fm implement OAuth in their API in future.

The result is exfm-oauth-api-proxy - currently deployed to exfmproxy.socialsignin.org - a simple webapp which delegates requests for a select subset of Ex.Fm’s API methods to Ex.Fm itself, but which also translates OAuth2 authenticated requests into username/password encoded requests before proxying the requests onwards.

This proxy is currently being used to support the “login/connect with exfm” use-cases at the following sites:

api.cloudplaylists.com

socialsignin.org

The exfm-oauth-api-proxy webapp was created from the sparklr2 sample in the spring-security-oauth project and provides the following OAuth2 endpoints that can be used to configure spring-social-exfm so it can be used within Spring Social enabled applications:

exfm.oauthTokenUrl=/oauth/token
exfm.oauthAuthorizeUrl=/oauth/authorize

exfm.oauthApiBaseUrl=/api/v3

Client Ids, Secrets and Redirect Uris are configured within the

oauth:client-details-service

element in spring-config.xml

The Java software engineer interview process

An experienced, qualified and enthusiastic builder is looking for his next job.

He is a builder that takes pride in the quality of his work and has found that this has pleased his previous clients who have all subsequently offered him follow-on projects.

The builder spots an advert online - a potential new client has a requirement to rebuild some of their walls - it’s a large project, maybe 6 months in duration, and they need a builder. No detail is given about the project other than the skills the builder must possess to do the job.

The builder does have the listed skills and he responds to the advert, sending through details of his previous clients, his online portfolio, and details of his accreditation (all of which is independently verifiable).

The client suggests a time he can call the builder to discuss the project further.  When the time comes,  the client introduces himself, and immediately starts to ask the builder the following questions:

“What is the best way to build walls?”

“Are all bricks concrete?

“What is the difference between bricks and mortar”

The builder is taken aback and bemused but answers the client’s questions politely.  To his surprise the questions keep coming, maybe another dozen or so.  After 15 minutes the client stops asking questions and asks if the builder has any of his own.  The builder replies:

“Perhaps you could give me some detail about the project you advertised online”

The client tells the builder that the project is to replace some existing walls but that the builder will have the opportunity to find out more should he be invited to quote. The phone call ends with the builder not really any the wiser about the project specified in the ad.

A short while later an invitation to quote is sent to the builder.  Attached to the invite are details of a task that the builder must complete before attending the quote - a requirement to build some very elementary walls out of plain old bricks and mortar. When complete, evidence of this task completion must be sent to the client where it will be assessed and discussed at the forthcoming quote.

By now the builder is pretty confused. He definitely remembered submitting details of his online portfolio showcasing his work to the client even before the initial phone call.

“I’m a builder”, he thinks to himself, “it’s what I do, I build walls built out of plain old bricks and mortar all the time.  There are examples of walls I’ve built clearly visible before on my online portfolio.”

“How could I have been worked with my previous clients if I didn’t know how to put together simple walls like this?”

“How could I have gained my accreditation with the company that created these very bricks we are using if I didn’t have this skill?”

Against his better judgement, the builder sighs and gets set to complete the walls task.  The task is simple to do, perhaps something like the walls he first learned to build in his youth, but it is boring and takes some time to complete - almost half a day in total.  “This project had better be worth bidding for” the builder thinks, shaking his head in bewilderment as he sends through evidence of his completed walls to the client.

To be continued…

Switch the context of this story to be that of a software engineer interviewing for a contract and this unlikely story becomes all too familiar…

Just replace the following words in the story to change context:

builder -> software engineer
bricks -> Java classes
mortar -> interfaces
walls -> software
quote -> interview


A Month Of Saturdays Playlist : March 2013

socialsignin.org

Check out http://socialsignin.org for a working deployed version of the socialsignin showcase webapp.

This webapp demonstrates using spring-social-security to delegate authentication to third party providers.

Other modules showcased are:

spring-social-lastfm

spring-social-soundcloud

spring-social-exfm

spring-social-mixcloud

spring-social-cloudplaylists

SocialSignin provider wrappers are used around the corresponding Spring Social modules to provide auto-registration of Spring Social components through component scanning and to provide an abstraction layer allowing the application to deal with API clients for a number of common use-cases instead of interacting with lower-level connection API

Users can create accounts with http://socialsignin.org by logging in with one of the available providers and once logged in can additionally connect their accounts with the remaining providers.

opensource.cloudplaylists.com

The original beta version of cloudplaylists.com was really a prototype, an experimental site that I used as a proof of concept for a new playlist sharing service that I could use to demo my work with Spring Social.

At the time, I hadn’t yet launched api.cloudplaylists.com, or written the spring-social-cloudplaylists client module, and as a consequence the original CloudPlaylists code was pretty bloated, with code spanning MVC layer and persistence deployed within the same application.

Since launching the API I wanted to re-launch the site, using a different model where the website code was much more lightweight, providing a UI on top of the functionality provided by the API.

I’ve deployed an initial version of this much more lightweight web app to http://opensource.cloudplaylists.com

This is a simple web application with skeleton HTML code, using spring-social-security and spring-social-cloudplaylists modules to delegate authentication, business logic and persistence to the CloudPlaylists API

This revamped site provides additional functionality over and above the original beta site, such as integration with Ex.Fm.

The intention is to open-source the entire code base for this site down the line - for now I’ve open-sourced the very basic skeleton HTML for the site on GitHub with the hope that someone with the right skill set can style this site to look beautiful.

spring-social-cloudplaylists

Spring Social module for CloudPlaylists - acting as a Java API client and allowing Spring-Social applications to provide “login with CloudPlaylists” functionality

api.cloudplaylists.com

Opening up the API for CloudPlaylists

cloudplaylists.com

The original beta version of CloudPlaylists - a site allowing playlists of SoundCloud tracks to be created quickly and easily, showcasing connectivity to SoundCloud and Last.Fm and delegating authentication to Spring-Social-Security to provide “login with SoundCloud” and “login with Last.Fm” functionality