Friday, December 25, 2009 1:06 AM bart

More LINQ with System.Interactive – Getting Started

With the recent release of the Reactive Extensions for .NET (Rx) on DevLabs, you’ll hear quite a bit about reactive programming, based on the IObservable<T> and IObserver<T> interfaces. A great amount of resources is available on Channel 9. In this series, I’ll focus on the dual of the System.Reactive assembly, which is System.Interactive, providing a bunch of extensions to the LINQ Standard Query Operators for IEnumerable<T>. In today’s installment we’ll talk about getting started with System.Interactive, also touching briefly on the deep duality.

 

Where to get it?

To get the Reactive Extensions, which include System.Interactive, visit the landing page on DevLabs over here. Downloads are available for .NET Framework 3.5 SP1, .NET Framework 4.0 Beta 2 and Silverlight 3. In this series, I’ll be using the “desktop CLR” distributions from Visual Studio 2008 and Visual Studio 2010.

The differences between the various distributions are of a technical nature and have to do with backporting certain essentials Rx relies on, to the .NET Framework 3.5 SP1 stack. For instance, the IObservable<T> and IObserver<T> interfaces exist in .NET 4.0 but don’t in .NET 3.5. Similarly, the Task Parallel Library (TPL) is available in .NET 4.0’s System.Threading namespace, while Rx redistributes it to run on .NET 3.5 SP1.

 

What’s in it?

Once you’ve installed, have a look at your Program Files (x86) folder, under Microsoft Reactive Extensions. I’m using the “DesktopV2” version here, which refers to CLR 2.0 and the .NET Framework 3.5 SP1 package. The main difference with the “DesktopV4” version is the presence of System.Threading, which contains the Parallel Extensions that ship in .NET 4.0:

image

A brief introduction to the remaining assemblies:

  • System.CoreEx.dll contains some commonly used types like Action and Func delegates with bigger arities (up to 16 parameters), new Property<T> primitives, a Unit type, an Event type wrapping “object sender, EventArgs e” pairs, a Notification<T> (which will be discussed extensively) and some notions of time in the form of TimeInterval<T> and Timestamped<T>.
  • System.Interactive.dll, the subject of this new series, contains extension methods for IEnumerable<T> and additional LINQ to Objects operators, provided in a type called EnumerableEx.
  • System.Reactive.dll, which is where Rx gets its name for and which will be discussed in future series, is the home for reactive programming tools. It contains IObservable<T> and IObserver<T>, as well as various combinators over it (sometimes referred to as “LINQ to Events”). In addition, it provides primitives like subjects and contains a join library (more about this in a separate installment).

 

Duality? Help!

As we like to use expensive words like “mathematical dual” it makes sense to provide some easy to grasp introduction to the subject. The first thing to look at is the distinction between interactive and reactive programming. In the diagram below, this is illustrated:

image

In the world of interactive programming, the application asks for more information. It pulls data out of a sequence that represents some data source, in particular by calling MoveNext on an enumerator object. The application is quite active in the data retrieval process: besides getting an enumerator (by calling GetEnumerator on an enumerable), it also decides about the pace of the retrieval by calling MoveNext at its own convenience.

In the world of reactive programming, the application is told about more information. Data is pushed to it from a data source by getting called on the OnNext method of an observer object. The application is quite passive in the data retrieval process: apart from subscribing to an observable source, it can’t do anything but reacting to the data pushed to it by means of OnNext calls.

The nice thing about those two worlds is that they’re dual. The highlighted words in the paragraphs above have dual meanings. Because of this observation, it’s desirable to search for dualities on a more formal and technical level as well. In particular, the interfaces being used here are the exact duals of one another: IEnumerable<T> is to IObservable<T> as IEnumerator<T> is to IObserver<T>. Dualization can be achieved by turning inputs (e.g. method parameters) into output (e.g. return values):

image

Lots of dualities exist in various disciplines, providing for great knowledge transfers between different domains. For example, in formal logic, De Morgan’s law allows converting expressions built from conjunctions into ones built from disjunctions, and vice versa. In electronics, similarities exist between the behavior of capacitors and inductances: know one and how to go back and forth between domains, and you know the other. Fourier calculus provides duals between time and frequency domains.

One thing all those have in common is a way to go back and forth between domains. Such a mechanism exists in the world of System.Reactive and System.Interactive as well. Every observable collection can be turned into an enumerable one and vice versa, using operators called ToEnumerable and ToObservable. To get a feel about how those work, imagine an enumerable collection first. The only thing one can do to retrieve its data is enumerate over it. For all the values received, signal them on the resulting observable’s observer. In the opposite direction, you subscribe on an observable collection to receive the values thrown at you and keep them so that the resulting enumerable can fetch them.

In this series, we’ll not look over the garden wall to the reactive world just yet. Instead, we’ll get our hands dirty in the world of System.Interactive, a logical extension to .NET 3.5’s IEnumerable<T> extension methods, known as the Standard Query Operators.

 

Operators overview

The System.Linq.EnumerableEx static class in System.Interactive contains various (extension) methods that operator on IEnumerable<T> enumerable collections. It should be seen as a logical extension to the System.Linq.Enumerable class in System.Core. In the illustration below I’ve summarize the various categories those new operators fall into. Some could be considered to fall in multiple categories, so take this with a grain of salt. Nevertheless, we’ll look at those big buckets in subsequent posts in this series:

  • Imperative use – provides operators that execute a sequence (Run) and inject side-effecting Actions in a chain of query operator calls (Do), which is handy for debugging.
  • Exceptions – enumeration of sequences can cause exceptions (e.g. if you write an iterator, but also by other means – see later), which may need to be handled somehow. Methods like Catch, Finally, Using, OnErrorResumeNext and Retry provide means to make a sequence resilient in face of exceptions.
  • Constructors – instead of creating an iterator yourself, it’s possible to let the system create a sequence on your behalf, e.g. by providing it a generator function (Generate), by composing sequences and elements (Return, StartWith, Throw), or triggering the call of a deferred constructor function when a client start enumerating (Defer).
  • Code = Data – the triplet of OnNext, OnError and OnComplete seen on IObserver<T> is a very code-centric way of signaling various outcomes of data consumption. An alternative view is to treat those outcomes as pieces of data, called notifications (Notification<T>). Using Materialize and Dematerialize, one can transfer back and forth between those two domains.
  • Combinators – producing sequences out of one or more existing sequences is what combinators generally do. One can repeat a sequence a number of times (Repeat), zip two sequences together (Zip), let two sequences battle to provide a result the fastest (Amb), and more. Those operators are most “in line” with what you already know from System.Linq today.
  • Functional – while the imperative and exception categories acknowledge the possibility for sequence to exhibit side-effects, the functional category is meant to tame the side-effects, typically in one-producer-many-consumer scenarios. When a sequence may produce side-effects during iteration, it may be desirable to avoid duplication of those when multiple consumers iterate.
  • Miscellaneous – just that, miscellaneous.

image

Next time, we’ll start by looking at the “Imperative use” category. Download the libraries today and start exploring!

Del.icio.us | Digg It | Technorati | Blinklist | Furl | reddit | DotNetKicks

Filed under: ,

Comments

# Twitter Trackbacks for More LINQ with System.Interactive ??? Getting Started - B# .NET Blog [bartdesmet.net] on Topsy.com

Pingback from  Twitter Trackbacks for                 More LINQ with System.Interactive ??? Getting Started - B# .NET Blog         [bartdesmet.net]        on Topsy.com

# re: More LINQ with System.Interactive – Getting Started

Friday, December 25, 2009 6:40 PM by Dax

I still don't get how it's a dual.  IEnumerable-IEnumerator is a one-to-many relationship, while IObservable-IObserver is many-to-many.  I also don't get why it matters.  Does IList have a dual, and if so, is it useful?

# re: More LINQ with System.Interactive – Getting Started

Friday, December 25, 2009 7:45 PM by bart

Hi Dax,

The IObservable-IObserver relationship is also one-to-many. Given one event source, one can have many subscriptions.

The reason it matters is, agreed, more for academic satifsaction though it acts as a good catalyst for design: if you find a new great operator on one side, chances are big it applies on the other side as well. In other words, it's a win-win situation.

Concerning IList<T>, sure we can find a dual. Let's call it TsilI<T>:

interface TsilI<T>

{

   int this[T t] { get; set; }

   T FoXedni(int i);

   Pair<int, T> Tserni();

   int TaEvomer();

}

So far, pure categorical hocus pocus. Now that the mathematician has left us, we should ask ourselves what the meaning could be.

What would it mean to index a T? Maybe it means that given element needs to be searched for, to get its index. Plausible. On the setter side, it could mean to insert it at that position. Great the dual of the indexer seems to be the original IList<T>'s IndexOf and InsertAt operations.

IndexOf turned into something that takes an int and produces a T; we know that already, it's the old indexer's get operation.

Insert and Remove are the most interesting ones as they're not as straightforward as the other ones. So far, it looks the dual to IList<T> is itself as we found the indexer and IndexOf operations back, as well as Insert by means of the dual to the indexer. What the duals to Insert and Remove look like should, intuitively, reflect the opposite meaning of "Hey, put that thing there" and "Hey, get rid of that thing there". One idea would be to allow the outside to query what has been inserted into the collection (opposite of Insert would be index, value pairs for all elements).

In such a setting you could almost fantasize about an index in an output position to stand for "I tell you there should exist an index such that":

- for Insert: there should be an element in there after the call

- for Remove: the element that's in there should be gone after the call

Dualizing "there exists" could mean to switch to "for any". And instead of "I tell you", it'd become "Please tell me". What we get now is "Please tell me any index for which":

- for Tserni: there's an element

- for evomeR: there was an element

Looks like the dual to insert would return a dictionary pairing up indices with elements, and the dual to remove would clear the list, returning all indexes (or an aggregate there of, like a count) that were removed.

Alternatively, one could see IList<T> as being too flattened, in particular as the aggregate of IIndexable<T> (containing the indexer and the IndexOf operation we dualized straightforwardly before) and IUpdatableByIndex<T>. It seems that the former dualizes into itself, where the latter could be turned into a notification mechanism. If IIndexable<T> were read-only, the IList<T> would provide a getter to obtain the updatable view on the list:

interface IList<T> : IIndexable<T>

{

   IUpdatableByIndex<T> GetUpdatable();

}

(In effect, we've made the object model collections' AsReadOnly call return object the default and introduce an AsWritable, in favor of a default immutability state.)

In this setting, the dual would turn into something that will receive notifications about updates, hence Insert and Remove would become OnInsert and OnRemove on the dual of IUpdatableByIndex<T>. Now we got observable collection types back, like ObservableList<T>.

While we started with a scientific approach, doing blind dualization, the remaining work is more of an art, often revealing semantics that were hidden from the signatures or interface declarations.

(No, I'm not providing dualization services for any interface in the BCL. See each of them as individual brain teasers.)

Thanks,

-Bart

# re: More LINQ with System.Interactive – Getting Started

Sunday, December 27, 2009 10:10 PM by Dax

Hmm, I still have to say I feel like semantically, the real dual of IEnumerable would be ILoadable or something like that; an "array" that can be loaded by LoadNext(T t) and Finish().  IObservable, to me, seems like something simply different, not necessarily dual.

Ultimately I'm still not convinced that the duality isn't broken in the design anyway.  An enumerable class and its enumerator are tightly coupled; at least the enumerable will know the implementation of IEnumerator returned by the GetEnumerator method.  However, IObserver and IObservable are loosely coupled.  And I still argue that the latter is a many-to-many relationship, for precisely that reason.  An observer can be loaded into any type of observable, and a single observer can watch many observers simultaneously.  And vice versa.  Whereas IEnumerable-IEnumerator is one-to-many by definition.  That definitely seems to break the semantic duality, but doesn't it break the categoric duality as well?

I like your analysis of the IList dual.  Sure you don't want to investigate the dual of RadioButton?  :)

# re: More LINQ with System.Interactive – Getting Started

Monday, December 28, 2009 2:31 AM by bart

Hi Dax,

Informally, duality is established by reversing arrows and compositions. So, let's simplify the use of interfaces and go with straight functions instead. Flatting IEnumerable<T> results in the following function:

() -> (() -> T)

Func<Func<T>>

We'll ignore the fact MoveNext can return false or can throw an exception, as well as the disposable nature of the enumerator. You could restore the full MoveNext behavior by using a discriminated union or whatnot. Just for clarity: the inner Func represents MoveNext and the outer one GetEnumerator.

Dualizing this would result in the following signature, reading the original from right to left:

(T -> ()) -> ()

Action<Action<T>>

This time, the inner Action represents the OnNext method, and the outer one does represent the Subscribe method.

In fact, this even correctly follows the expected change in variance, since Func<T> is covariant in T and Action<T> is contravariant in T.

As a result, I'd argue it's expected you observe different arities in the relationships between participating entities. What has truly changed because of the arrow reversal and variance swap is the fan-in and fan-out of the interfaces. From a user's point of view, an enumerable is a fan-out interface: you get an enumerator in return and that's forced upon you by the enumerable implementation. In the world of observable, you're in the other seat, providing your callback (fan-in) observer object to the observable.

This is all as a I would expect.

Thanks,

-Bart

PS: Purely in terms of behavior, a RadioButton is a realization of the "there exists one" operator, with regards to the group it operates in. You can be sure one and only one option will be selected. I'll leave the math to the reader to think about what a reasonable dual may be.

# The Morning Brew - Chris Alcock &raquo; The Morning Brew #506

Tuesday, December 29, 2009 3:57 AM by The Morning Brew - Chris Alcock » The Morning Brew #506

Pingback from  The Morning Brew - Chris Alcock  &raquo; The Morning Brew #506

# Social comments and analytics for this post

Tuesday, December 29, 2009 7:07 AM by uberVU - social comments

This post was mentioned on Twitter by marcocz: (xmax) .NET System.Interactive http://bit.ly/7icZFh

# Reactive Extensions for .NET (Rx) &laquo; Just Justin&#039;s

Saturday, February 06, 2010 3:45 AM by Reactive Extensions for .NET (Rx) « Just Justin's

Pingback from  Reactive Extensions for .NET (Rx)  &laquo; Just Justin&#039;s

# Interactive Extensions for .NET or &quot;Ix&quot; &laquo; Mehran&#039;s NotePad

Pingback from  Interactive Extensions for .NET or &quot;Ix&quot; &laquo;  Mehran&#039;s NotePad

# Reactive Programming ????????????????????????????????????

Pingback from  Reactive Programming ????????????????????????????????????