Michael Lavelle

“Reinforcement learning (RL) and supervised learning are usually portrayed as distinct methods of learning from experience. RL methods are often applied to problems involving sequential dynamics and optimization of a scalar performance objective, with online exploration of the effects of actions. Supervised learning methods, on the other hand, are frequently used for problems involving static input-output mappings and minimization of a vector error signal, with no explicit dependence on how training examples are gathered. As discussed by Barto and Dietterich (this volume), the key feature distinguishing RL and supervised learning is whether training information from the environment serves as an evaluation signal or as an error signal…”

[Source :  Supervised Actor-Critic Reinforcement Learning - paper discussing problems where both kinds of feedback ( evaluation signal/error signals ) are available to a learning system at the same time) ]

“Having discussed key differences between reinforcement learning and supervised learning, the question arises as to whether these differences are fundamental or merely superficial differences that can be eliminated with suitable problem reformulation.”

[ Source :
Reinforcement Learning and its Relationship to Supervised Learning ]

Further reading:

Reinforcement Learning Architectures

Supervised Actor-Critic Reinforcement Learning

Generalized Model Learning for Reinforcement Learning in Factored Domains

Autonomous Helicopter flight via Reinforcement learning

Real-time reinforcement learning in continuous domains

Reinforcement learning for rule extraction from a labeled dataset

Reinforcement Learning by Comparing Immediate Reward

Maximum reward reinforcement learning: A non-cumulative reward criterion

http://www.compapp.dcu.ie/~humphrys/PhD/ch2.html

http://webdocs.cs.ualberta.ca/~sutton/book/ebook/node7.html

http://pages.cs.wisc.edu/~finton/what-rl.html

These notes are my own observations and questions from my study of the material on Stanford’s Engineering Everywhere Machine Learning Course

Machine Learning :  a scientific discipline concerned with the design and development of algorithms that allow machines (computers) to make decisions or even evolve behaviors based on collection of empirical data.

The algorithms of machine learning fall into the following primary categories ( source : Wikipedia ) :

• Unsupervised Learning :  the problem of trying to find hidden structure in unlabeled data

• Supervised Learning : inferring a function from labeled training data

• Reinforcement Learning : concerned with how an agent ought to take actions in an environment so as to maximize some notion of cumulative reward

( There are other classifications of algorithms, however these three classes are the major types discussed in the reference material I’ve encountered so far  )

Supervised vs. Unsupervised Learning

We can see that Supervised Learning and Unsupervised Learning are mutually exclusive categories if the training data is either all labelled or all unlabelled.

What does it mean to say data is labelled ?

Alternative definitions of the Supervised and Unsupervised learning from a Stanford University glossary help us see what it means to say data is labelled:

• Supervised Learning:  Learning techniques used to learn the relationship between independent attributes and a designated dependent attribute (the label)

• Unsupervised Learning : Learning techniques that group instances without a pre-specified dependent attribute.

We can say from these definitions that training data is said to be labelled when the data being analysed is comprised of pairs consisting of a set of independent input attributes and a dependent output attribute (the label), whereas with unlabeled data there is no pre-specified dependent attribute.

When a dataset is unlabeled, we may wish to use an Unsupervised Learning algorithm to find patterns within the structure of this data.

When a dataset contains labels, the label is often referred to as the “correct” or desired answer, and may wish to use a Supervised Learning algorithm to infer a function which predicts this “right answer” given a set of input attributes.

“To describe the Supervised Learning algorithm slightly more formally, our goal is, given a training set, to learn a function h : X → Y so that h(x) is a “good” predictor for the corresponding value of y. For historical reasons, this function h is called a hypothesis.”

( here X denotes the space of input values, and Y the spaces of output values.)

[Source: CS229 Lecture Notes - 1 ]

( The terms predict and predictor are highlighted in the above text as we will shortly attempt to define what we mean by these terms in more detail. )

Examples of Unsupervised/Supervised Learning

If we had a dataset consisting of the location coordinates of houses within a given city, we could say this is an unlabelled data set and we could use an Unsupervised Learning algorithm to find clusters of homes in the city.  

If our dataset included the price of each house in addition to the location attributes, we could use a Supervised Learning algorithm to look for a function which predicted the price of the house given its location.

Where does Reinforcement Learning fit in - is it Supervised or Unsupervised Learning?

Reinforcement Learning : concerned with how an agent ought to take actions in an environment so as to maximize some notion of cumulative reward

Craig Walls talks at SpringOne 2GX 2012 about Spring Social.

Check out the “Spring Social Project Family” slide.  Feel proud that my Last.Fm, SoundCloud and Mixcloud modules are listed as members :-)

exfm-oauth-api-proxy : A proxy to Ex.Fm’s API supporting OAuth2 Authentication - currently deployed to exfmproxy.socialsignin.org

(source code at https://github.com/socialsignin/exfm-oauth-api-proxy )

Some time ago I wanted to provide “login with Ex.Fm” functionality on a number of my websites and to access authenticated methods of Ex.Fm’s API on behalf of my logged in users.

I started work on a spring-social-exfm module that acts as a Java client, but as Ex.Fm’s API does not currently support OAuth this module was limited in terms of it’s compatibility with the Spring Social framework, requiring a user to explicitly provide a username and password to access authenticated API methods.

I started experimenting with creating a proxy to Ex.Fm’s API that supports OAuth2 authentication, which would then mean I could develop a Spring Social module for Ex.Fm which could be used as an OAuth client should Ex.Fm implement OAuth in their API in future.

The result is exfm-oauth-api-proxy - currently deployed to exfmproxy.socialsignin.org - a simple webapp which delegates requests for a select subset of Ex.Fm’s API methods to Ex.Fm itself, but which also translates OAuth2 authenticated requests into username/password encoded requests before proxying the requests onwards.

This proxy is currently being used to support the “login/connect with exfm” use-cases at the following sites:

api.cloudplaylists.com

socialsignin.org

The exfm-oauth-api-proxy webapp was created from the sparklr2 sample in the spring-security-oauth project and provides the following OAuth2 endpoints that can be used to configure spring-social-exfm so it can be used within Spring Social enabled applications:

exfm.oauthTokenUrl=/oauth/token
exfm.oauthAuthorizeUrl=/oauth/authorize

exfm.oauthApiBaseUrl=/api/v3

Client Ids, Secrets and Redirect Uris are configured within the

oauth:client-details-service 

element in spring-config.xml

An experienced, qualified and enthusiastic builder is looking for his next job.

He is a builder that takes pride in the quality of his work and has found that this has pleased his previous clients who have all subsequently offered him follow-on projects.

The builder spots an advert online - a potential new client has a requirement to rebuild some of their walls - it’s a large project, maybe 6 months in duration, and they need a builder. No detail is given about the project other than the skills the builder must possess to do the job.

The builder does have the listed skills and he responds to the advert, sending through details of his previous clients, his online portfolio, and details of his accreditation (all of which is independently verifiable).

The client suggests a time he can call the builder to discuss the project further.  When the time comes,  the client introduces himself, and immediately starts to ask the builder the following questions:

“What is the best way to build walls?”

“Are all bricks concrete?

“What is the difference between bricks and mortar”

The builder is taken aback and bemused but answers the client’s questions politely.  To his surprise the questions keep coming, maybe another dozen or so.  After 15 minutes the client stops asking questions and asks if the builder has any of his own.  The builder replies:

“Perhaps you could give me some detail about the project you advertised online”

The client tells the builder that the project is to replace some existing walls but that the builder will have the opportunity to find out more should he be invited to quote. The phone call ends with the builder not really any the wiser about the project specified in the ad.

A short while later an invitation to quote is sent to the builder.  Attached to the invite are details of a task that the builder must complete before attending the quote - a requirement to build some very elementary walls out of plain old bricks and mortar. When complete, evidence of this task completion must be sent to the client where it will be assessed and discussed at the forthcoming quote.

By now the builder is pretty confused. He definitely remembered submitting details of his online portfolio showcasing his work to the client even before the initial phone call.

“I’m a builder”, he thinks to himself, “it’s what I do, I build walls built out of plain old bricks and mortar all the time.  There are examples of walls I’ve built clearly visible before on my online portfolio.”

“How could I have been worked with my previous clients if I didn’t know how to put together simple walls like this?”

“How could I have gained my accreditation with the company that created these very bricks we are using if I didn’t have this skill?”

Against his better judgement, the builder sighs and gets set to complete the walls task.  The task is simple to do, perhaps something like the walls he first learned to build in his youth, but it is boring and takes some time to complete - almost half a day in total.  “This project had better be worth bidding for” the builder thinks, shaking his head in bewilderment as he sends through evidence of his completed walls to the client.

To be continued…

Switch the context of this story to be that of a software engineer interviewing for a contract and this unlikely story becomes all too familiar…

Just replace the following words in the story to change context:

builder -> software engineer
bricks -> Java classes
mortar -> interfaces
walls -> software
quote -> interview

Check out http://socialsignin.org for a working deployed version of the socialsignin showcase webapp.

This webapp demonstrates using spring-social-security to delegate authentication to third party providers.

Other modules showcased are:

spring-social-lastfm

spring-social-soundcloud

spring-social-exfm

spring-social-mixcloud

spring-social-cloudplaylists

SocialSignin provider wrappers are used around the corresponding Spring Social modules to provide auto-registration of Spring Social components through component scanning and to provide an abstraction layer allowing the application to deal with API clients for a number of common use-cases instead of interacting with lower-level connection API

Users can create accounts with http://socialsignin.org by logging in with one of the available providers and once logged in can additionally connect their accounts with the remaining providers.

Spring Social module for CloudPlaylists - acting as a Java API client and allowing Spring-Social applications to provide “login with CloudPlaylists” functionality

I’ve open-sourced an initial cut of spring-social-exfm - a Java client for Ex.Fm’s API.

Spring Social modules can normally be registered within a Spring Social enabled webapp which is then able to perform the OAuth authentication dance with an API - passing back a session token after user approval which can then be used for subsequent requests which require authentication.   This means that it’s the third party website which collects username/password details from a user and the client application is only aware of access tokens.

Unfortunately at the time of writing, Ex.Fm’s API doesn’t support OAuth - - instead operations which require authorization on behalf of a particular user require username and password to be sent as request parameters.

The consequence of this is that In terms of using this module to interact with Ex.Fm’s API directly, this module can only really be used as a standalone Java client, and can’t be fully “plugged-in” to Spring Social enabled webapps in the same way as for other providers.

ExFm exFm = new ExFmTemplate("http://ex.fm/api/v3"."username","password");
ExFmProfile profile = exfm.meOperations().getUserProfile();

However, In the event that Ex.Fm do support an OAuth2 authentication for their API in future, I’ve added support for this to the spring-social-exfm module and have created a simple oauth proxy to Ex.Fm’s API for showcase purposes which I’ve used to test the module.  ( source code at https://github.com/socialsignin/exfm-oauth-api-proxy )

I’ve provided an ExFmConnectionFactory which can be constructed with client keys, secrets and the auth endpoints provided by this proxy ( or with the auth endpoints of an OAuth API from Ex.Fm should one be implemented in the future).

public ExFmConnectionFactory(String clientId, String clientSecret,String oauthAuthorizeUrl, String oauthTokenUrl,String oauthApiBaseUrl) {...}

spring-social-security : an add-on for spring-social webapps, achieving sign-in-with-provider capability directly within the Spring Security filter chain, delegating authentication for your application to SaaS providers.

Spring Social provides a number of controllers which can be used within Spring MVC webapps to handle authorization flow with SaaS providers - namely ConnectController and ProviderSignInController.

These controllers assume that there is the concept of a local user within your web application - in other words that your web application is using some sort of security framework.

When I first started using the spring social framework I found myself having to set up Spring Security in my application as a pre-requesite of enabling users to connect to third party providers. 

In my first attempts, this meant using an authentication filter such as UsernamePasswordAuthenticationFilter,  writing a UserDetailsService etc - and the result was that “login with provider xyz” functionality ended up being an option that sat alongside standard “login with username/password”.  The Spring Security framework had no awareness of the Spring Social implementation - when requesting protected resources the user would be faced with a username/password login page and I was unable to specify that I would like the user to connect with a certain provider in order to access a particular resource.

In fact, what I wanted to achieve was almost the reverse of this set up - I wanted users to be able to authenticate with a third party provider, and for this third party authentication to drive the authentication in my application.  I wanted to protect resources in my application based on the users’ authentication status with a third party - for example to be able to specify access to  certain resource requires authentication with Twitter, or perhaps with Last.Fm.  I didn’t want to provide a username/password store for my application - instead I wanted to do away with the concept of a local username/password entirely and use a third party’s authentication functionality to identify users.  

I wanted to address these concerns and so I set about creating and open-sourcing Spring Social Security  -  the aim of the module is to put Spring Social at the centre of a site’s security configuration, to make the Security configuration of a Spring Social web application easy to set up and fully aware of the connectivity a user has with third party providers.

Resources can be protected based on whether the user has connected with a external account from a specified third party provider.

<intercept-url pattern="/sendTweet" 
access="hasRole('ROLE_USER_TWITTER')" />

<intercept-url pattern="/scrobble" 
access="hasRole('ROLE_USER_LASTFM')" />

Provider-specific access denied handling is enabled so as to prompt users to connect with the required provider in order to gain access to the resource.

The Spring-Social-Security module comes with a set of out-of-the-box components which can be registered by simple component scanning, and by adding the SpringSocialSecurityAuthenticationFilter to the Spring Security filter chain.

Also provided is a default SpringSocialSecuritySignUpController which uses the UsersConnectionRepository provided by Spring Social to store details of registered users.  This behaviour can be customised if required in order to save registered users details in a separate user details store - this is discussed further in this blog post.

For more details on using this module please see Spring Social Security on GitHub.

2012:

1987:

Spring Social requires that your application provides implementations of ConnectionRepository and UsersConnectionRepository - these components are used for the persistence of users third party API connection details.

There are JDBC versions that ship with spring-social-core, the JdbcConnectionRepository and the JdbcUsersConnectionRepository which need a table to exist in your database created by with a structure such as that provided by JdbcUsersConnectionRepository.sql.

I’ve needed to use a couple of alternative implementations over the past months and thought I’d share them on GitHub.  

spring-social-roo-connectionrepository 

spring-social-hibernate-jpa     ( Hibernate implementation of spring-social-jpa )

There are also two in-memory implementations embedded within other projects I’ve worked on - InMemoryUsersConnectionRepository and QuickstartUsersConnectionRepository which could help when setting up a project quickly in development mode.

To test the implementations I’ve created an abstract test harness which can be subclassed by developers writing their own ConnectionRepository components to help with testing.  AbstractUsersConnectionRepositoryTest is a essentially a modified version of JdbcUsersConnectionRepositoryTest from spring social core - examples of implementation-specific subclasses in use can be found in any of the above named projects.

By default Spring-Social-Security ships with a set of components which handle the creation of new local accounts for users of your application when they first connect with a third party provider. 

These components are found within the “signup” package and are registered by default component scanning.  With the “signup” package excluded from component scan, these default components could be explicitly registered as follows:

<context:component-scan base-package="org.socialsignin.springsocial.security" >
  <!-- Exclude signup components from Spring Social Security  -->
  <context:exclude-filter type="regex" expression=".*\.signup\..*"/>
</context:component-scan>

    <bean  
class="org.socialsignin.springsocial.security.signup.ConnectionRepositorySignUpService" /> 

    <bean  
class="org.socialsignin.springsocial.security.signup.SpringSocialSecuritySignUpController" />
 
    <bean  
class="org.socialsignin.springsocial.security.demo.CustomUserProfileFactory" /> 

    <bean  
class="org.socialsignin.springsocial.security.signup.SpringSocialSecurityConnectionSignUp" />

To customise signup behavior, replace any or all of these beans with your own custom implementations. 

For an example of an custom signup configuration, where registered users’ details are stored in a separate User table using JPA, please see the following fork of the spring-social-security-demo project

https://github.com/michaellavelle/spring-social-security-demo

The latest release of SocialSignin’s Spring-Social-Security allows resources to be protected based on whether the user has connected with a external account from a specified third party provider.

<intercept-url pattern="/sendTweet" 
access="hasRole('ROLE_USER_TWITTER')" />

<intercept-url pattern="/scrobble" 
access="hasRole('ROLE_USER_LASTFM')" />

Provider-specific access denied handling prompts users to connect with the required provider in order to gain access to the resource.

I’m loving Ex.Fm and wanted to import my SoundCloud favorites into my Ex.Fm profile as loved tracks.   Spent this afternoon making some changes to spring-social-exfm and spring-social-soundcloud modules to allow me to do this with a few lines of Java code.

Here’s a snippet from my new spring-social-music sample utils module that does the job…

// Construct a new SoundCloudTemplate only authorized by api key, as user-level authorization is not needed
// to retrieve favorite tracks
SoundCloud soundCloud = new SoundCloudTemplate(soundCloudApiKey);
		
// Construct a new ExFmTemplate authorized for a specific user, specifying api base url, username and password
ExFm exFm = new ExFmTemplate("http://ex.fm/api/v3",exFmUserId,exFmPassword);
		
// Retrieve the first page of SoundCloud favorite tracks
Page<Track> soundCloudFavorites = soundCloud.usersOperations().userOperations(soundCloudUserId).getFavorites(new PageRequest(0,soundCloudFetchPageSize));
		
// For each page of SoundCloud favorite tracks...
for (int pageNumber = 0; pageNumber < soundCloudFavorites.getTotalPages();pageNumber++)
{
	// For each SoundCloud favorite track on current page, love the track on Ex.Fm using the stream url
	for (Track soundCloudTrack : soundCloudFavorites)
	{
		sleepForLimitRate();
		try
		{
			exFm.songOperations().loveSongBySourceUrl(soundCloudTrack.getStreamUrl());
		}
		catch (ResourceNotFoundException e)
		{
			// Failed to love song as Ex.Fm can't find information about the soundcloud track with this url
		}
	}
        // Get next page of tracks from SoundCloud if available
        if (soundCloudFavorites.hasNextPage()) soundCloudFavorites = soundCloud.usersOperations().userOperations(soundCloudUserId).getFavorites(new PageRequest(pageNumber + 1,soundCloudFetchPageSize));
}

Checkout the spring-social-music module to see the full source code and dependencies used by the snippet above, or add the following repository and dependency to your maven project in order to use or modify the simple utility class for your project.

	<repositories>
             <repository>
		<id>opensourceagility-snapshots</id>
		<url>http://repo.opensourceagility.com/snapshots</url>
             </repository>
	</repositories>
	<dependencies>
          <dependency>
            <groupId>com.springsocialmusic</groupId>
            <artifactId>spring-social-music</artifactId>
           <version>1.0.1-SNAPSHOT</version>
        </dependency>
</dependencies>

The utility class to import SoundCloud favorites into Ex.Fm as loved tracks could then be called as shown below:

SoundCloudToExFmMigrator soundCloudToExFmMigrator = new SoundCloudToExFmMigrator(mySoundCloudApiKey);
		soundCloudToExFmMigrator.loveSoundCloudFavoritesOnExFm(fromSoundCloudUserName, exFmUserName,exFmPassword);