Higher Kinds in C# with language-ext [Part 10- ReaderT monad transformer]

This is a multi-part series of articles – and you'll need to know much of what's in the earlier parts to follow this – so it's worth going back to the earlier episodes if you haven't read them already:

• Part 1 - Introduction, traits, semigroups, and monoids
• Part 2 - Functors
• Part 3 - Foldables
• Part 4 - Applicatives
• Part 5 - Validation
• Part 6 - Traversables
• Part 7 - Monads
• Part 8 - Monads (continued)
• Part 9 - Monad Transformers

In the last article we started looking into monad transformers. We mostly focused on the MaybeT transformer (OptionT in language-ext), lifting the IO monad into it to create an optional side-effect.

Later in the article I did a quick overview of a project (that I am working on) that creates 'domain monads' by wrapping monad-transformer stacks into new types that are then further composed with free-monads to create a small application architecture.

There is never a 'right way' to structure an application and I think it's risky to use blogs, or other short-form methods to share coding advice on the 'one true way', because it doesn't exist. However, I think when you see how domains, layers, components, and structure in-general, evolves out of compositional types (like monads and monad-transformers) then that light-bulb moment will happen where you can suddenly 'see' the structure of your architecture from 10,000 feet.

These moments of clarity were always missing for me in my former OOP life, whereas with the pure functional programming approach I feel I have total clarity, but also it feels like the architecture is fluid and malleable (it can change with requirements) in a way that just isn't there in the object-oriented world. This may well be a personal thing, so please don't take this as me saying "there's exactly one way to write code" – because all self-appointed internet experts really should be taken with a handful of salt!

Even though the architecture example (from the last article) was a small example, I hope that it whetted your appetite when thinking about how to use pure functional programming techniques to create structure in your programs!

To really benefit the most from this approach we should aim to make our entire code-base pure and getting the architecture right is part of that story. What I find valuable about this approach is that the architecture itself becomes composable. That's quite profound when you think about it and it certainly changes the way I write code now.

It was quite a lot to take in, of course, and I certainly wouldn't expect novice FP devs to get it straight-away. But, one of the biggest issues I've had with building language-ext has been how to put control of building functional types into the hands of the users of the library. Before the higher-kinds approach (demonstrated in this series) there was no way to write code that worked for all monads, or all functors, etc. Now you can.

And before we had monad-transformers there was no way to compose the monads that already existed into new 'super monads', which meant I was stuck writing OptionAsync, EitherAsync, TryOption, and other combination-monads. And you were stuck waiting for me to build various pre-combined monads. This issue no longer exists.

These abstractions now exist. And many of the core types needed for you to build your own functional stacks now exist. The days of waiting for me to build ValidationAsync<F, A> are over, just use the ValidationT<F, IO, A> monad transformer stack.

It may take a little while for the implications of the changes in v5 of language-ext to really materialise as opportunities for you. It's a style of programming and an approach to composition that just hasn't existed in C# to-date. So, unless you've come from a Haskell background, this might not seem to be quite as profound as it really is.

C# isn't Haskell of course, but the reason for all of this isn't to make C# into Haskell, it's to make the inertia of the C# developer track in the direction of pure functional programming. And it just so happens that Haskell (as a pure functional programming language) has solved many patterns we need and so the inspiration often comes from that language.

Let's take a look at another monad-transformer...

ReaderT<Env, M, A>

The ReaderT monad-transformer adds a static environment, Env, to a given monad M. We briefly covered the Reader<Env, A> monad in the Monads Continued article – this generalises the idea into a monad-transformer. So, for example, let's say we had some application configuration in a simple record type:

public record MyConfig(string SourceFolder);

We want to write pure code, so we can't just stick this in a global static variable.

One option is to pass it as an argument to every function that needs the configuration settings:

public static IO<string> readSourceText(string file, MyConfig config)
{
    var path = Path.Combine(config.SourceFolder, file);
    return IO.liftAsync(e => File.ReadAllTextAsync(path, e.Token));
}

This is perfectly valid, but it can start to get a bit messy and annoying. And, aside from the name of the type, it's not very declarative. If MyConfig had a less descriptive name then it might make the interface to the function more confusing.

Instead, we drop the MyConfig argument and use the ReaderT monad-transformer. We can stack it with the IO monad to get both a static environment and side-effects.

ReaderT will automatically thread the MyConfig value through the computation for us and do so declaratively:

public static ReaderT<MyConfig, IO, string> readSourceText(string file) =>
    from config in ReaderT.ask<IO, MyConfig>()
    let path = Path.Combine(config.SourceFolder, file)
    from text in IO.liftAsync(e => File.ReadAllTextAsync(path, e.Token))
    select text;

Notice how there isn't a MyConfig value passed as an argument to readSourceText. It somehow magics the configuration out of thin air!

The key to this is on the first line in the LINQ expression:

ReaderT.ask<IO, MyConfig>()

This static function: ask, somehow gets the MyConfig structure and loads it into the config variable. How can a static function retrieve a non-global state?

If we look at the definition of ReaderT<Env, M, A> we'll get some clues:

public record ReaderT<Env, M, A>(Func<Env, K<M, A>> runReader) : K<ReaderT<Env, M>, A>
    where M : Monad<M>, SemiAlternative<M>
{
    public K<M, A> Run(Env env) => 
        this.runReader(env);   

   ...    
}

Internally, ReaderT is a Func<Env, K<M, A>>. If we provide an Env then we get a K<M, A> back (which is constrained to be a Monad<M>).

There's also a SemiAlternative<M> constraint. Ignore that for now, I'll cover it in a later article.

This follows the transformer pattern. Except this is a little different to MaybeT and OptionT in that K<M, A> is the inner type, whereas with MaybeT and OptionT it was the outer type: K<M, Option<A>>.

This difference in implementation is all to do with what is best for the particular transformer. It makes sense to provide an environment for the inner monad to run in. Whereas, for the optional types, it makes sense for the result to be optional! In general, optional types are wrapped by the lifted monad, and stateful-computation types wrap the lifted monad.

When we come to run the ReaderT transformer, we must pass the configuration value to it, so it can be passed through the computation:

// Create the application configuration object.  
// Typically you would do this once
var configObj = new MyConfig("c:\\folder");

// Run the ReaderT.  This returns an IO<string>
var configTextIO = readSourceText("test.cs").Run(configObj);

The result is an IO<string>. The IO monad being that which was lifted into the transformer.

If we run the IO value then we get the raw text:

var configText = configTextIO.Run();

Reminder: You shouldn't exit the IO monad unless you're at the very edge of your application.

That doesn't explain how ask works from earlier. It somehow magicked the configuration out of thin air. Remember, we're working with a wrapped function here, which means we can defer the gathering of the configuration (make it lazy):

new ReaderT<Env, M, Env>(env => M.Pure(env));

So, we don't need to know the environment to implement ask, we just need to know we will be given it at some point. Here we take the env input and simply lift it into the M monad which exposes it as the bound value. Extremely simple, but also extremely effective.

It can be simplified further to:

new ReaderT<Env, M, Env>(M.Pure);

And even further when it's part of a function that can infer the return type:

public static ReaderT<Env, M, Env> ask<Env, M>() 
    where M : Monad<M>, SemiAlternative<M> =>
    new (M.Pure);

Everything in ReaderT therefore is lazy. We know the Run method will provide an Env value which we can use to invoke the encapsulated delegate and realise a concrete result. But, every other (non-Run) method on ReaderT must use standard function composition to achieve its behaviours.

For example, the Bind function:

public ReaderT<Env, M, B> Bind<B>(Func<A, ReaderT<Env, M, B>> f) =>
    new(env => M.Bind(runReader(env), x => f(x).runReader(env)));

We create a new Func that provides us with an Env env, we invoke our runReader function with the env value. It gives us the wrapped K<M, A> which we can M.Bind because it's constrained to be a Monad<M>. That gets us the A value from within the M monad (assigned to x). We pass the A to the f function provided. It returns a new ReaderT which we then invoke (with the same env). It returns a K<M, B> which is what M.Bind expects as a result!

This is how the environment is threaded through the monad-transformer and how it can be made available for use in the LINQ expression. The env is always available because every ReaderT has it given when invoked.

Readable

So, that's all awesome and everything, but when we use transformers it's often good to wrap up the stack into a new type for encapsulation reasons. The eagle eyed among you may realise that ReaderT<Env, IO, A> is essentially Eff<RT, A> from previous versions of language-ext. The Eff monad was designed to be an IO monad with a 'runtime' that allowed for injectable side-effects.

So, let's wrap it up and create our own simplified Eff monad. First, let's create the Eff<RT, A> type:

public record Eff<RT, A>(ReaderT<RT, IO, A> runEff) : K<Eff<RT>, A>;

This simply wraps the ReaderT<RT, IO, A> monad transformer.

Next, let's make Eff<RT, A> into a monad by implementing the Monad trait:

public class Eff<RT> : Monad<Eff<RT>>
{
    public static K<Eff<RT>, B> Bind<A, B>(K<Eff<RT>, A> ma, Func<A, K<Eff<RT>, B>> f) =>
        new Eff<RT, B>(ma.As().runEff.Bind(x => f(x).As().runEff));

    public static K<Eff<RT>, B> Map<A, B>(Func<A, B> f, K<Eff<RT>, A> ma) =>
        new Eff<RT, B>(ma.As().runEff.Map(f));

    public static K<Eff<RT>, A> Pure<A>(A value) =>
        new Eff<RT, A>(ReaderT.Pure<RT, IO, A>(value));

    public static K<Eff<RT>, B> Apply<A, B>(K<Eff<RT>, Func<A, B>> mf, K<Eff<RT>, A> ma) =>
        new Eff<RT, B>(mf.As().runEff.Apply(ma.As().runEff).As());

    public static K<Eff<RT>, A> LiftIO<A>(IO<A> ma) =>
        new Eff<RT, A>(ReaderT.liftIO<RT, IO, A>(ma));
}

Note how every method is just a wrapper around the calls to the ReaderT implementations (runEff is a ReaderT<RT, IO, A>). Also note that we have implemented LiftIO which isn't enforced (there's a default implementation) but this allows us to use IO operations in our LINQ expressions directly (we covered this in the last article).

Next, create our As and Run extensions and add some constructors:

public static class Eff
{
    public static Eff<RT, A> As<RT, A>(this K<Eff<RT>, A> ma) =>
        (Eff<RT, A>)ma;

    public static IO<A> Run<RT, A>(this K<Eff<RT>, A> ma, RT runtime) =>
        ma.As().runEff.Run(runtime).As();

    public static Eff<RT, A> Pure<RT, A>(A value) =>
        new (ReaderT.Pure<RT, IO, A>(value));

    public static Eff<RT, A> Fail<RT, A>(Error error) =>
        new (ReaderT.liftIO<RT, IO, A>(IO.Fail<A>(error)));
}

And that's it!

Of course, the real Eff monad comes with lots of helper functions that make working with effects easier. But pretty much everything is just lifting the existing functionality from the transformers into the Eff type. For example, to access the runtime is simply a ReaderT.ask:

public static Eff<RT, RT> runtime<RT>() =>
    new (ReaderT.ask<IO, RT>());

You should be getting the idea now that this is just a wrapper around the transformer stack. We could extend the capabilities by stacking more transformers if necessary.

The real reason I wanted to demonstrate this is because there is another trait-type especially for readers. It's called: Readable and any type that implements it becomes 'readable':

public interface Readable<M, Env>
    where M : Readable<M, Env>
{
    public static abstract K<M, A> Asks<A>(Func<Env, A> f);

    public static virtual K<M, Env> Ask =>
        M.Asks(Prelude.identity);

    public static abstract K<M, A> Local<A>(
        Func<Env, Env> f,
        K<M, A> ma);
}

For those who know Haskell, this is essentially MonadReader without inheriting the Monad constraint!

We can add it to the traits that Eff<RT> implements:

public class Eff<RT> : Monad<Eff<RT>>, Readable<Eff<RT>, RT>
{
    // ... implementations from earlier ...

    public static K<Eff<RT>, A> Asks<A>(Func<RT, A> f) =>
        new Eff<RT, A>(ReaderT<RT, IO, A>.Asks(f));

    public static K<Eff<RT>, A> Local<A>(Func<RT, RT> f, K<Eff<RT>, A> ma) =>
        new Eff<RT, A>(ma.As().runEff.Local(f));
}    

Readable just abstracts the idea of accessing a static environment away from the monad. It means we can write code that accepts any 'readable' type, whether it's Eff, ReaderT, Reader, or whatever bespoke type we have built.

Readeable also comes with some static functions that makes readable types easy to work with:

public static class Readable
{
    public static K<M, Env> ask<M, Env>()
        where M : Readable<M, Env> =>
        M.Ask;

    public static K<M, A> asks<M, Env, A>(Func<Env, A> f)
        where M : Readable<M, Env> =>
        M.Asks(f);

    public static K<M, A> asksM<M, Env, A>(Func<Env, K<M, A>> f)
        where M : Readable<M, Env>, Monad<M> =>
        M.Flatten(M.Asks(f));

    public static K<M, A> local<M, Env, A>(Func<Env, Env> f, K<M, A> ma)
        where M : Readable<M, Env> =>
        M.Local(f, ma);
}

Let's say our 'runtime' was just an int and we want to write a function that takes two Eff monads with int as their bound-values and then sum those values with the int that's in the runtime (the environment).

public static K<Eff<int>, int> addEff(Eff<int, int> mx, Eff<int, int> my) =>
   from x in mx
   from y in my
   from z in Eff.runtime<int>()
   select x + y + z;

This works just fine, we can access the runtime value via the Eff.runtime function. However, this function only works with one monadic type: Eff.

We can generalise it even further:

public static K<M, int> addRdr<M>(K<M, int> mx, K<M, int> my)
    where M : Readable<M, int>, Monad<M> =>
    from x in mx
    from y in my
    from z in Readable.ask<M, int>()
    select x + y + z;

Note how M is not only constrained to be a Monad<M> but it is also constrained to be a Readable<M, int>. So, it's bindable and readable! Let's try running it with some Eff monads:

var compute = addRdr(Eff.Pure<int, int>(100), Eff.Pure<int, int>(200));
var result = compute.Run(300).Run();  // 600

addRdr has no idea what monad we're running and it has no idea how we're managing our static environment. It is a complete abstraction away from the internals. The Eff monad supports IO side-effects, parallelism, resource tracking and auto-cleanup, error tracking, retries, repeats, ..., and a static environment. All of which could happen in the addRdr function (because mx and my are computations) – but the implementation doesn't care, it is abstracted away. All it cares about is that the values are bindable and readable!

This is the path to true abstraction – where everything is pure. We can safely abstract, because we know all of the components are playing by the rules!

You may think this kind of abstraction isn't that valuable. But, if you remember the real-world domain-monads example from the previous article – where I created a Db monad, a Service monad, and an Api monad – then it's not a massive stretch to imagine writing some very general support functions that work with all of those monads (using the Monad constraint) and accessing their configuration (using the Readable constraint).

For example, you could imagine a User value being in the environment and a getLoggedInUser function to access it from any monad that supports it:

public record User(string Id);
public record Session(User LoggedIn, DateTime LastAccess);

public static K<M, Session> getSession<M>()
    where M : ReaderM<M, Session> =>
    ReaderM.ask<M, Session>();

public static K<M, User> getLoggedInUser<M>()
    where M : ReaderM<M, Session> =>
    ReaderM.asks<M, Session, User>(s => s.LoggedIn);

We could expand that out to embed security roles and rights so we could do inline assertions throughout our code-base:

// Simple User type with list of roles it's a member of
public record User(string Id, Seq<Role> Memberships);

// Union type of rights
public abstract record Right; 
public record CanViewDemographic : Right;
public record CanSendInvoice : Right;

// Role with its allowed rights
public record Role(Seq<Right> Rights);

Next we can write the function exactly once to deal with access-rights:

public static K<M, Unit> assertHasRight<M, R>()
    where R : Right
    where M : Readable<M, Session>, Functor<M> =>
    getLoggedInUser<M>().Map(
        u => u.Memberships.Exists(
            role => role.Rights.Exists(
                right => right is R))
            ? Prelude.unit
            : throw new SecurityException("Access denied"));

Then at the start of an API call, for example, you could setup the User in a Session environment (pre-populating the rights and roles once per request). That Session can then be passed to the computation that represents the API request (via Run(session)) which will then automatically thread the session, user, and rights through your monadic stack.

The (read-only) user, security rights, and roles will automatically propagate through your code. Each monad that wants to have access to security credentials could just expose them via a Readable trait implementation.

We can then write our security assertions code in any monad that supports them:

public static Eff<Session, Unit> sendInvoice() =>
    from _ in assertHasRight<Eff<Session>, CanSendInvoice>()
    from r in doSendInvoice()
    select r;

Being able to write security code once and keep it tightly audited is the mark of a professional approach to software development. So these abstractions are not just academic examples of what can be done with pure functional programming, they're very real examples of how you can make your code more robust and more secure. You'll be the auditors best friend – although I'm not sure that's a good thing!

Local environments

Having a fixed static environment for your entire code-base is of course possible, but that will tend to expose more 'environment' than is necessary for any subsystem. And so, it would be good if we could swap out our larger environments for something smaller on-the-fly. This is where ReaderT.local(f, ma) and ReaderT.with(f, ma) comes in:

• local maps an Env to an Env – this allows simple mutation of the environment. The ma computation is then run in that locally mutated environment. By the end of the call the environment is reset.
• with maps an Env to an alternative type which can be any other value you like, but often it is derived from Env. For example, you may have an AppConfig which contains a DbConfig – it could make sense to only expose the DbConfig to your data-layer by mapping from AppConfig -> DbConfig.

Note that with is not available in the Readable trait and so you must use Reader.with and ReaderT.with to map the environment to a different type (meaning we can't fully generalise the environment mapping). For any type that encapsulates ReaderT or Reader it is worth adding your own equivalent of with to do environment mapping (if that makes sense for your type, some wrappers might have a fixed environment). This is a limitation brought about by how the higher-kinds are implemented (the Env is baked into the trait implementation) – I haven't thought of an easy solution to this yet!

Conclusion

I often show (and will continue to show) the IO monad being the monad that is lifted into the transformer. I'm mostly doing this because it's easier to see tangible examples of what it means to compose monads with transformers. Just remember that any monad can be lifted into ReaderT – it could be a Validation<F, A> to have an environment for your validators (for example).

One other thing to note is that ReaderT would usually be the outermost type in the transformer stack. This is usually because you want the inner monad(s) in the transformer stack to have access to the environment. This isn't a rule, it's just usually the most effective position.

The ReaderT transformer really is the first 'control' transformer that shows how capabilities can be 'bolted on' to existing monads in a way that starts to break away from the relatively 'small thinking' of optional types and other common FP topics. Being able to carry a – pure – environment through your code really is the first part of building a pure architecture.

In previous versions of language-ext we only have Reader which was just an environment carrying monad – nothing else, it didn't compose with anything and that meant it was probably never used (I certainly never used it). Now we have the ability to augment with readable traits and that is extremely powerful!

Next up will be the StateT transformer. Very much like the ReaderT transformer except the environment can be modified. Pure mutation? You bet!

Part 11 - StateT monad transformer