Jef Claes on software and life

Validating composite models with knockout validation

2013-05-20T17:02:00.000+02:00

When you use knockout validation to extend observables with validation rules, it will add a few functions to these observables - the most important ones being; error and isValid. You can use these functions to verify if any of the validation rules were violated, and to extract an error message.

To extract all of the error messages out of a composite model, you can use a grouping function.

function BookingModel() {      
    var self = this;
   
    self.contact = new ContactModel();
    self.departure = new DepartureModel();
   
    self.isValid = function() {
        return self.contact.isValid() && self.departure.isValid();
    };
   
    self.validate = function() {                           
        if (!self.isValid()) {
            var errors = ko.validation.group(self);                           
            errors.showAllMessages();
        
            return false;          
        }
        return true;
    };                    
}
function DepartureModel() {
    this.street = ko.observable('').extend({ required: true });
    this.houseNumber = ko.observable('').extend({ required: true });
    this.city = ko.observable('').extend({ required: true });
    this.time = ko.observable('').extend({ required: true });            
    
    this.isValid = function() {
        return 
            this.street.isValid() &&
            this.houseNumber.isValid() &&
            this.city.isValid() &&
            this.time.isValid();                
    };
}
function ContactModel() {
    this.firstName = ko.observable('').extend({ required: true });
    this.lastName = ko.observable('').extend({ required: true });
    this.phoneNumber = this.firstName = ko.observable('').extend({ required: true });
    this.email = ko.observable('').extend({ required: true });   
    this.isValid = function() {
        return 
            this.firstName.isValid() &&
            this.lastName.isValid() &&
            this.phoneNumber.isValid() &&
            this.email.isValid();                
    };    
}

This is what my first attempt looked like. A little later I discovered that you can get rid of these boilerplate functions on the composite model by applying the validatedObservable function instead.

ko.applyBindings(ko.validatedObservable(new BookingModel()));
ko.validatedObservable = function (initialValue) {
    if (!exports.utils.isObject(initialValue)) { 
        return ko.observable(initialValue).extend({ validatable: true }); }
    var obsv = ko.observable(initialValue);
    obsv.errors = exports.group(initialValue);
    obsv.isValid = ko.computed(function () {
        return obsv.errors().length === 0;
    });
    return obsv;
};

The validatedObservable function will add an errors function to the composite model which traverses the object graph and validates each eligible property. The errors function also has a showAllMessages function that will display an error message next to each invalid element. The isValid function only asserts if there are any errors.

function BookingModel() {      
    var self = this;
    
    self.contact = new ContactModel();
    self.departure = new DepartureModel();
    self.validate = function() {                           
        if (!self.isValid()) {                                         
            self.errors.showAllMessages();
        
            return false;          
        }
        return true;
    };             
}   
function DepartureModel() {
    this.street = ko.observable('').extend({ required: true });
    this.houseNumber = ko.observable('').extend({ required: true });
    this.city = ko.observable('').extend({ required: true });
    this.time = ko.observable('').extend({ required: true });            
}
function ContactModel() {
    this.firstName = ko.observable('').extend({ required: true });
    this.lastName = ko.observable('').extend({ required: true });
    this.phoneNumber = this.firstName = ko.observable('').extend({ required: true });
    this.email = ko.observable('').extend({ required: true });        
}

Removing that cruft results in less bulky, cheaper models.

If you try this example, you will notice that the model appears to be valid even though the validation rules are clearly violated. It took me a few minutes of browsing the source to figure out why this was happening. When you use the group functions to validate your model, they will by default only look at first level properties. So if you have a composite model, you need to modify the grouping validation configuration, and set the deep property to true.

ko.validation.init({ grouping : { deep: true, observable: true } });

IDDD Tour notes (2/2)

2013-05-12T15:44:00.000+02:00

This is the second and last part of my notes I scribbled down attending the IDDD Tour. The first part was published last week.

A better model

Even if you come up with a better model, the fact that it has been the ubiquitous language of the domain for decades proves that it works for them.

This quote bothers me a bit. There definitely is truth to this, but modeling an existing process often presents such a great opportunity to revise and improve it. Naked models don't conceal deficiencies, inefficiencies and aberrations. Exploring alternative models free of habituation, politics and legacy is dirt cheap, while the outcome could considerably benefit all. It seems such a shame not to take advantage of this. As with most things, know when to pick your fights.

Elegance

Elegance is for dressing, not for delivering software.

This is one to remember; I'll be using this one next time someone uses elegance as an argument for gold plating.

Cultivating models

Models grow; you will never have the best model from the start. Improve them every time you pass by.

Your first attempt at it is hardly ever right. Don't beat yourself up over it. The best models are the result of multiple iterations.
In general I consider dwelling on one problem too long to be a waste of time. Settle for good enough, and allow the better solution to emerge by itself over time.
When I feel a design is mighty important, I might accelerate this process by iterating over it multiple times in just a few days; asking for constant feedback, carrying the problem with me everywhere I go, always challenging it, and molding it in different shapes until one sticks.

Reuse

Lots of people use a shared kernel just for reuse. It's often not worth it.

People are so obsessed with the DRY principle, and the dogma of avoiding duplication, it often does more harm than good. Nonexistent concepts are introduced just to spare a few duplicate lines of code, while they will hinder and complicate autonomous evolution.

REST and DDD

Don't expose your domain model over REST: expose use cases.

You want to enable relentlessly evolving your domain model without breaking clients. In practice you would post intent-revealing command resources, while hypermedia guides you in navigating to subsequent commands.

Published language

How is the language agreed upon; over lunch or by a committee?

A great question which reveals if a language should be really considered as published.

And that's the last of it. If you thought some of these were interesting, you should probably get the book. I finished Growing Object-Oriented Software Guided by Tests last week, so I finally get to read it myself after having it collect dust for two months.

IDDD Tour notes (1/2)

2013-05-05T17:10:00.000+02:00

Two weeks ago I got to spend four days attending the IDDD Tour by Vaughn Vernon. Although my book queue has only allowed me to shallowly browse the book, I had high hopes for this course. I anticipated a week of getting lectured on DDD with a few practical exercises, but was blown away by the openness and interaction promoted by Vaughn and his associate Alberto Brandolini. A passionate group, engaging workshops, long days and lots of sharing made these few days exceptionally satisfying and inspirational. I'm grateful to those who got this show on the road; it was more than worth your trouble.

Over these few days I scribbled down some things I thought were worth remembering or worth some more thought. Most of these are aimed at the strategic side of DDD and the softer side of software since that's where I acquired most new insights.

Complexity

If we really understood the complexity of a system, we would very often make different decisions from when designing new systems.

Understanding the complexity of a new system is hard. Modeling can serve as the perfect tool to help you expose the hidden complexities as early as possible.
Often a system appears to be not much more than CRUD, to quickly evolve into something with a lot more behaviour than anticipated. Make sure you don't sabotage growing your design by picking an architecture or framework that doesn't allow you to.
I often start with just exposing commands and queries to my application, which encapsulate all domain logic. Behind these command- and queryhandlers is very little abstraction, just some basic building blocks; aggregates, entities and value objects. Later on I can relentlessly refactor everything behind these commands and queries without breaking application code: introduce domain services, events, separate the write- from the read model etc...

Change

How do you bring in DDD? Gather a small cluster of motivated silent people and work on a project that has lots of potential value.

This is my recipe for change as well. Find a few like-minded people that are confident drawing outside the lines set out by the status quo, work in the shades, working your way towards the light.

Brown

There is always brown, even when it's under the green.

Every application has parts which are less pretty, even green fields; use proper encapsulation to contain them, and avoid them corrupting other parts of the system.

Another brick in the wall

Break down that paper wall between domain experts and developers. Learn the favorite coffee of the domain expert.

Lots of organizations still build this paper wall between domain experts and developers. But paper doesn't answer to questions, nor does it carry nuances very well. Communicating on a more personal level is key to tearing down that wall. Breaking the ice can be as easy as treating your domain expert to coffee.

One language

Which language do you use when naming things in code? Yours.

I don't think native English speakers understand just how awkward it is to write code in something other than English. Keywords, API's, documentation... it's all in English. I feel that putting these together with your own language results in a cacophony of words.
In my current project the business is bilingual; they use Dutch and French names for the same concept interchangeably. We tried to find middle ground by introducing English names, and have a map between these languages. Ideally there would only be one language though.
Someone mentioned that they use English for all globally unambiguous concepts such as account number, name... but use their mother tongue in scenarios where nuances would be lost in translation. This seems to be an attractive compromise.

Defining a bounded context

A bounded context is where one ubiquitous language is consistent.

This is by far the simplest definition of a bounded context.

Bounded context relations

You say please, so I am upstream.

The relationship between bounded contexts is expressed by saying one is upstream or downstream. I thought of the quote above to be a creative expression that can aid in explaining these two concepts.

Rewrites

When you rewrite an application, you have to look at the existing forces in an organization; chances are the previous team wasn't terrible either.

Tons of factors influence the outcome of a project; it's not just about teams and their skillset. Some forces in an organization can make it close to impossible to ship good software; fear of change, lack of vision, bone-dry information streams... So don't assume your rewrite will fix everything.

Surprise

Adding surprise value is often not that valuable.

This seems to be related to the mantra of 'good enough'. Sometimes we - with the best intentions - invest more time and money in solving a problem in the best possible way, than it will ever return.

These were some notes I wrote down the first two days. I'll try to go through the other half over these next few days. I hope you found some of them valuable too.

Customizing a knockout validation message

2013-04-28T18:56:00.000+02:00

Knockout validation is a plugin that allows you to extend observables with validation. If you're familiar with jQuery validation, you will notice that most provided validation rules are very similar.

One of the validation rules available out-of-the-box is pattern validation. By default the error message for violating this rule doesn't mention the expected pattern. And in general, showing the expected pattern probably isn't a good idea since your users will have no idea what you're talking about. If you're building a back-end for devops purposes, it might be more acceptable though.

Instead of "Invalid.", I want my error message to be "I can't believe the pattern doesn't match {pattern}.".

test('1. When the object is invalid, the error message contains our formatted message.', 
    function () {
        var testObj = ko.observable('').extend({ pattern: 'myPattern' });    
            
        testObj('something');
        
        equal(
            testObj.error, 
            'I can\'t believe the pattern doesn\'t match \'myPattern\'.', 
            'The formatted error message is correct.');    
    }
);
Expected: "I can't believe the pattern doesn't match 'myPattern'."
Result:    "Invalid."
Diff: "I can't believe the pattern doesn't match 'myPattern'." "Invalid."

Digging a bit into the source, I found the function that formats the error messages; it simply replaces {0} in the message with the params value.

formatMessage: function (message, params) {
    if (typeof (message) === 'function')
        return message(params);
    return message.replace(/\{0\}/gi, params);
};

So to change the error message to have it include the pattern, we can pass a pattern object which contains our message with the params placeholder and the params value.

test('2. When the object is invalid, the error message contains our formatted message.', 
    function () {   
        var testObj = ko
            .observable('')
            .extend({ pattern: {
                message: 'I can\'t believe the pattern doesn\'t match \'{0}\'.',
                params: 'myPattern'
        }});                            
                
        testObj('something');
        
        equal(
            testObj.error, 
            'I can\'t believe the pattern doesn\'t match \'myPattern\'.', 
            'The formatted error message is correct.');    
    }
);
The formatted error message is correct.
Expected: "I can't believe the pattern doesn't match 'myPattern'."

This will make our test pass. It's not a great idea to repeat this in each location though. Since the plugin exposes its rules, you can just go in, modify the default message for the pattern rule, and see it applied everywhere.

test('3. When the objects are invalid, the error message contains our formatted message.', 
    function () {
        var testObj1 = ko.observable('').extend({ pattern: 'myPattern' });
        var testObj2 = ko.observable('').extend({ pattern: 'myPattern' });
                                            
        ko.validation.rules['pattern'].message = 
            'I can\'t believe the pattern doesn\'t match \'{0}\'.',    
        
        testObj1('something');
        testObj2('something');
        
        equal(
            testObj1.error, 
            'I can\'t believe the pattern doesn\'t match \'myPattern\'.', 
            'The formatted error message is correct for testObj1.');      
        equal(
            testObj2.error, 
            'I can\'t believe the pattern doesn\'t match \'myPattern\'.', 
            'The formatted error message is correct for testObj2.');      
    }
);
The formatted error message is correct.
Expected: "I can't believe the pattern doesn't match 'myPattern'."
The formatted error message is correct.
Expected: "I can't believe the pattern doesn't match 'myPattern'."

I hope that even if you weren't interested in this exact scenario, this post gave you an idea on how to go at customizing knockout validation messages.

Modeling the four-eye principle

2013-04-21T19:48:00.000+02:00

Working in a financial domain over the last year, it was only a matter of time before I would be confronted with one of the variations of the two-man rule: the four-eye principle. Satisfying the principle is simple enough; an extra pair of eyes needs to approve of requested changes before they're applied to the system. This measure should prevent mistakes such as a user nuking North Korea by accident, or transferring all corporate funds to a personal off-shore bank account. In practice all you need is an accomplice.

Although I have seen artifacts of this concept, this is the first time I actively had to model it myself. Since I looked online for inspiration, but returned empty handed, I'm documenting my findings here.

I started out modeling the concept directly on the relevant entities, but it was obvious really soon that this was going nowhere. The solution headed in the wrong direction from the get go: copying records, dirty tracking, a bunch of possible states, resulting in complex transitions, and the need for an audit log. I couldn't even address these concerns without them bleeding into other layers such as queries and repositories. This design also failed to cover one edge scenario where we don't need an entity at all.

Returning home, I gave it another shot, but this time I modeled the principle as a separate concept, not touching any existing entities. A user hands in a request, which will be persisted, and will just sit there until a supervisor approves or declines it. If it's accepted, the request state gets updated, and all request handlers are invoked. It's the request handlers that are responsible for applying the requested change(s) to the system. Before the request handlers are invoked, no existing entities are touched.

Modeling the concept this way also leans closer to reality. You want to keep your core system as clean and simple as possible, to avoid confusion and mistakes where you can. If we would go back in time twenty years, the requests would also have lived in a separate drawer.

Here's some of my tinkering trying to support this concept.

A request is actually a template. It will always contain the current state, and some meta data concerning previous state transitions, but the actual content is generic. When a supervisor accepts or declines a request, a guard makes sure the user that made the original request, is not the one doing the supervision. Other guards make sure the status transition makes sense; you need to make a new request once it has been supervised. When the request is accepted, a domain event gets raised.

public class Request
{
    public Request(Requested requested, IRequestContent content)
    {
        Guard.ForEmpty(requested, "requested");
        Guard.ForEmpty(content, "content");
        Id = Guid.NewGuid();
        Requested = requested;
        Content = content;
        Status = Status.Waiting;
    }
    public Guid Id { get; private set; }
    public Requested Requested { get; private set; }
    public Supervised Supervised { get; private set; }
    public IRequestContent Content { get; private set; }
    public Status Status { get; private set; }
    public void Accept(Supervised supervised)
    {         
        ChangeStatus(Status.Accepted, supervised);
        DomainEvents.Raise(new RequestAcceptedEvent() { RequestId = Id });
    }
    public void Decline(Supervised supervised)
    {           
        ChangeStatus(Status.Declined, supervised);            
    }
    private void ChangeStatus(Status status, Supervised supervised)
    {
        Guard.ForEmpty(supervised, "supervised");
        Guard.SupervisingUserBeingTheRequestingUser(Requested.By, supervised.By);
        if (Status != Status.Waiting)
            throw new InvalidOperationException(
                "This request is already accepted or declined.");
        if (status == Status.Waiting)
            throw new InvalidOperationException(
                "The status shouldn't be changed to Waiting after creation.");            
        Status = status;
        Supervised = supervised;
    }
    public override bool Equals(object obj)
    {
        if (obj == null || GetType() != obj.GetType())
            return false;
        var other = obj as Request;
        return Id == other.Id;
    }
    public override int GetHashCode()
    {
        return this.Id.GetHashCode();
    }
}

An event handler listens for the request accepted event, and uses the content's type to make a generic type, which is then used to search the container for relevant request handlers. Since our type system isn't aware of our specific types - request handlers and request content are both objects at this point, I sprinkle some dynamic on top, and rely on the DLR to invoke the correct method. I think drawing outside the static lines here helps keeping things simple; it's definitely nicer to read than its counterpart, which needs to rely on reflection.

public interface IRequestHandler<TRequestContent>
{
    void OnAccepted(TRequestContent requestContent);
}
public class RequestAcceptedEventHandler : IEventHandler<RequestAcceptedEvent>
{
    private readonly IKernel _kernel;
    private readonly IRequestRepository _requestRepository;
    public RequestAcceptedEventHandler(IKernel kernel, IRequestRepository requestRepository)
    {
        _kernel = kernel;
        _requestRepository = requestRepository;
    }
    public void Handle(RequestAcceptedEvent @event)
    {
        var request = _requestRepository.GetById(@event.RequestId);
        var requestHandlerType = 
            typeof(IRequestHandler<>)
                .MakeGenericType(request.Content.GetType());
        foreach (var requestHandler in _kernel.GetAll(requestHandlerType))
            ((dynamic)requestHandler).OnAccepted((dynamic)request.Content);
    }
}

Now that we have this in place, a scenario where a user requests to have a missile launched, and a supervisor approves of this request, could look like this.

static void Main(string[] args)
{
    var repository = kernel.Get<IRequestRepository>();
    
    var launchMissileRequest = new Request(
        new Requested(new User("MiniMe")),
        new LaunchMissileRequestContent("V-2", "The Ozone Layer"));
    
    repository.Add(launchMissileRequest);
    ListRepositoryContent(repository);
    
    var request = repository.GetById(launchMissileRequest.Id);
    
    request.Accept(new Supervised(new User("DrEvil")));
    ListRepositoryContent(repository);
    Console.ReadLine();
}
public class LaunchMissileRequestHandler : IRequestHandler<LaunchMissileRequestContent>
{
    public void OnAccepted(LaunchMissileRequestContent requestContent)
    {    
        Console.WriteLine(string.Format("Launching {0} to {1}!!",
            requestContent.MissileName, requestContent.Destination));
    }
}
public class LaunchMissileRequestContent : IRequestContent
{
    public LaunchMissileRequestContent(string missileName, string destination)
    {
        Guard.ForEmpty(missileName, "missile name");
        Guard.ForEmpty(destination, "destination");
        MissileName = missileName;
        Destination = destination;
    }
    public string MissileName { get; set; }
    public string Destination { get; set; }
    public string Summary
    {
        get { return string.Format("Launch {0} to {1}.", MissileName, Destination); }
    }
}

This will result in the following being outputted to the console.

-----------------------------------------
Id: dcd6eaba-1a3e-4d81-82f4-4b2938911e7e
Status: Waiting
Summary: Launch V-2 to The Ozone Layer.
-----------------------------------------
Launching V-2 to The Ozone Layer!!
-----------------------------------------
Id: dcd6eaba-1a3e-4d81-82f4-4b2938911e7e
Status: Accepted
Summary: Launch V-2 to The Ozone Layer.
-----------------------------------------

Fully exposed

It's rare to see people share how they go at modeling real-life problems - money and withdrawing funds from an account don't count. And I get it; you're fully exposed; there is more than one way to skin a cat; history and context is lost in translation; and the devil is in the details. It's a shame though; there is no technology or methodology that will save you when you get these things catastrophically wrong.

The source of this article is up on GitHub. If you feel like toying with this concept, be my guest. Be gentle about it though.

Internationalization in NancyFx

2013-04-14T17:53:00.000+02:00

When I'm building web applications for non-work projects, I still find myself using Nancy most of the time. Something I looked at this week was how I could get the user's culture in Nancy. I looked a bit at the latest source, and discovered that recently some work has been done in this area.

When you look at the Nancy context, available in modules, you will find a Culture property.

public SomeModule()
{
    Get["/"] = _  => {
        var cultureInfo = Context.Culture;
        return View["Home.cshtml"];
    };
}

Tracing through the source, you can see that this value is set in the default contextfactory, which in its turn queries a culture service. The service itself iterates over all culture conventions until it finds one that returns a value.

public CultureInfo DetermineCurrentCulture(NancyContext context)
{
    CultureInfo culture = null;
    foreach (var convention in this.cultureConventions)
    {
        culture = convention(context);
        if (culture != null)
        {
            break;
        }
    }
    return culture;
}

The default culture conventions make use of some built-in culture detection mechanisms. Six mechanisms are in place, and will be tried in a specific order.

conventions.CultureConventions = new List<Func<NancyContext, CultureInfo>>(6)
{
    BuiltInCultureConventions.FormCulture,
    BuiltInCultureConventions.PathCulture,
    BuiltInCultureConventions.HeaderCulture,
    BuiltInCultureConventions.SessionCulture,
    BuiltInCultureConventions.CookieCulture,
    BuiltInCultureConventions.ThreadCulture
};

Form: does the form data contain a CurrentCulture entry?
Path: is the first part of the url a culture?
Header: is the Accept-Language header present?
Session: is there a CurrentCulture key in the session?
Cookie: is there a CurrentCulture key in the cookie?
Thread: what's the current thread culture?

So out-of-the-box Nancy is very smart about how she will try to determine a user's culture. You can either pass the culture to the relevant formatting functions, or you can set the culture for the execution of the whole request.

The latter can be achieved with a before hook. You can add this to an individual module, or to each module in your application by adding it to the bootstapper.

protected override void RequestStartup(
    IKernel container, 
    IPipelines pipelines, 
    NancyContext context)
{
    base.RequestStartup(container, pipelines, context);
    pipelines.BeforeRequest += ctx =>
    {
        Thread.CurrentThread.CurrentCulture = context.Culture;
        return null;
    };
}

I posted this on the mailing list and GrumpyDev raised one concern; depending on in which host you're running Nancy, it might not be guaranteed that the whole request is executed on the same thread, so that could produce some undesired results. ASP.NET runs each request on its own thread though.

Designing entities: immutability first

2013-04-07T17:35:00.000+02:00

The first year I wrote software for a living I spent my days mostly writing forms over data applications; most of my efforts were wasted just trying to make things work using ASP.NET and the Webforms engine. It was only after a year and graduating from the School of Hard Knocks that I learned there is a lot more to building clean and maintainable software than knowing the ins' and outs' of a proprietary UI technology.

One of the habits I have adapted over time, is designing new entities as immutable objects first, and to go from there. I believe this helps me to make more deliberate design decisions, leading to better designs.

Allow me to demonstrate this using a customer entity - I know this is cliché, but I have a hard time coming up with original examples.

A constructor

This one might sound rather obvious, but you would be surprised how many classes don't have the decency of a simple constructor, especially since C# introduced object initializers. While object initializers definitely look good, they make your code harder to maintain; classes don't communicate what they need to get by, nor do they get the opportunity to protect their invariants as early as possible.

Like I said, I design my classes to be immutable from the start; after initialization they can't be modified. I'm not even exposing any state for now.

public class Customer
{
    public Customer(
        Guid id,
        string firstName,
        string lastName,
        string street,
        string city,
        string country,
        DateTime birthdate,
        decimal balance)
    {
        FirstName = firstName;
        LastName = lastName;
        Street = street;
        City = city;
        Country = country;
        BirthDate = birthdate;
        Balance = balance;
    }
    public Guid Id { get; private set; }
    private string FirstName { get; set; }
    private string LastName { get; set; }
    private string Street { get; set; }
    private string City { get; set; }
    private string Country { get; set; }
    private DateTime BirthDate { get; set; }
    private decimal Balance { get; set; }
}

Comparing identities

An entity is always unique within a system; it has an identity. To encapsulate comparing two entities, and to avoid repeating myself, I always override the Equals and GetHashCode methods. We're not guided into doing this by our object's immutability though, but by common sense.

public override bool Equals(object obj)
{
    if (obj == null || GetType() != obj.GetType())
        return false;            
    var other = obj as Customer;
       
    return Id == other.Id;
}
public override int GetHashCode()
{
    return this.Id.GetHashCode();
}

Defaults

When a new customer is created, its balance is still empty. The constructor is the perfect place to initialize this default.

public Customer(
    Guid id,
    string firstName,
    string lastName,
    string street,
    string city,
    string country,
    DateTime birthdate)
{
    FirstName = firstName;
    LastName = lastName;
    Street = street;
    City = city;
    Country = country;
    BirthDate = birthdate;
    Balance = 0;
}

Protecting invariants

The next thing I'll do is use the constructor to guard for arguments that don't satisfy my objects invariants. This way we guarantee that the object can never be initialized with an invalid state; avoiding a bunch of trouble along the way.

We expect a customer to have a first name, last name, street, city and country which isn't empty. We also expect its birth date not to be in the future.

public Customer(
    Guid id,
    string firstName,
    string lastName,
    string street,
    string city,
    string country,
    DateTime birthdate)
{
    Guard.ForNullOrEmpty(firstName, "first name");
    Guard.ForNullOrEmpty(lastName, "last name");
    Guard.ForNullOrEmpty(street, "street");
    Guard.ForNullOrEmpty(city, "city");
    Guard.ForNullOrEmpty(country, "country");
    Guard.ForDatesInTheFuture(birthdate, "birthdate");
    Id = id;
    FirstName = firstName;
    LastName = lastName;
    Street = street;
    City = city;
    Country = country;
    BirthDate = birthdate;            
    Balance = 0;
}
public class Guard
{
    public static void ForNullOrEmpty(string value, string desc)
    {
        if (string.IsNullOrEmpty(value))
            throw new NullReferenceException(desc);
    }
    public static void ForDateInTheFuture(DateTime value, string desc)
    {
        if (value > DateTimeProvider.Now())
            throw new ArgumentException(
                string.Format("{0} can't be a date in the future.", desc))
    }
}

Composition and extracting value objects

I'm fairly confident that every self-respecting programmer is annoyed by the fact that the obvious concepts name and address haven't been extracted yet. Especially since that bloated constructor is all up in your face. Let's go ahead and extract two value objects: name and address.

public class Customer
{
    public Customer(
        Guid id,
        Name name,
        Address address,
        DateTime birthdate)
    {
        Guard.ForNull(name, "name");
        Guard.ForNull(address, "address");
        Guard.ForDateInTheFuture(birthdate, "birthdate");
        Id = id;
        Name = name;
        Address = address;
        Balance = 0;
    }
    public Guid Id { get; private set; }
    
    private Name Name { get; set; }
    private Address Address { get; set; }
    private DateTime BirthDate { get; set; }
    private decimal Balance { get; set; }
}
public class Name
{
    public Name(string first, string last)
    {
        Guard.ForNullOrEmpty(first, "first name");
        Guard.ForNullOrEmpty(last, "last name");
        First = first;
        Last = last;
    }
    private string First { get; set; }
    private string Last { get; set; }
    public string Full 
    {
        get
        {
            return string.Format("{0} {1}", First, Last);
        }
    }
    
    public override bool Equals(Object obj)
    {
        ...
    }
    
    public override int GetHashCode()
    {
        ...
    }
}
public class Address
{
    public Address(string street, string city, string country)
    {
        Guard.ForNullOrEmpty(street, "street");
        Guard.ForNullOrEmpty(city, "city");
        Guard.ForNullOrEmpty(country, "country");
        Street = street;
        City = city;
        Country = country;
    }
    private string Street { get; set; }
    private string City { get; set; }
    private string Country { get; set; }
    
    public override bool Equals(Object obj)
    {
        ...
    }
    
    public override int GetHashCode()
    {
        ...
    }
}

There, that looks better.

Tell, don't ask

In good OOP we tell objects what to do, we don't want to query an object for its state and do something based on that; that would lead to broken encapsulation and thereby more brittle code.

Since our immutable object isn't exposing any state yet, we are forced to ask ourselves why we want to expose something. This way, disregarding "tell don't ask" becomes a deliberate choice.

Let's say we want to decrease the balance, and throw an exception when funds are insufficient. Instead of asking the customer instance and acting on that..

if (!customer.HasSufficientFunds())
    throw new InsufficientFundsException();

It's better to encapsulate data and behaviour, and tell it what to do instead.

public class Customer
{
    public Customer(
        Guid id,
        Name name,
        Address address,
        DateTime birthdate)
    {
        Guard.ForNull(name, "name");
        Guard.ForNull(address, "address");
        Guard.ForDateInTheFuture(birthdate, "birthdate");
        Id = id;
        Name = name;
        Address = address;
        Balance = 0;
    }
    public Guid Id { get; private set; }
    
    private Name Name { get; set; }
    private Address Address { get; set; }
    private DateTime BirthDate { get; set; }
    private decimal Balance { get; set; }
    private bool HasSufficientFunds(decimal value) 
    {
        return Balance - value >= 0;
    }
    public void DecreaseBalance(decimal value)
    {
        if (HasSufficientFunds(value))
            throw new InsufficientFundsException();
        Balance -= value;
    }
}

Making operations explicit

Next thing we want to do is change the address of a customer. Since we're not exposing the address property directly, we're also forced to think about a useful name for the operation; I decided to name it MoveTo. With this we make things more explicit, and thus more evident for people who are new to the domain.

public class Customer
{
    public Customer(
        Guid id,
        Name name,
        Address address,
        DateTime birthdate)
    {
        Guard.ForNull(name, "name");
        Guard.ForNull(address, "address");
        Guard.ForDateInTheFuture(birthdate, "birthdate");
        Id = id;
        Name = name;
        Address = address;
        Balance = 0;
    }
    public Guid Id { get; private set; }
    
    private Name Name { get; set; }
    private Address Address { get; set; }
    private DateTime BirthDate { get; set; }
    private decimal Balance { get; set; }
    ...
    public void MoveTo(Address newAddress)
    {
        Guard.ForNull(newAddress, "new address");
        Address = newAddress;
    }
}
var customer = new Customer(
    Guid.NewGuid(),
    new Name("Jef", "Claes"),
    new Address("Ergens", "Antwerp", "Belgium"),
    new DateTime(1987, 10, 18));
customer.MoveTo(new Address("Quelque part", "Brussels", "Belgium"));

Opening up bit by bit

In the meanwhile our entity is no longer immutable, but we're doing a good job of not exposing any state. If you're using the same model for reads as you do for writes, you're going to need to expose some properties eventually; for querying, but also for assertions. Make sure to only expose the getters though, and to avoid violating "Tell, don't ask" since that became a lot easier now.

public class Customer
{
    public Customer(
        Guid id,
        Name name,
        Address address,
        DateTime birthdate)
    {
        Guard.ForNull(name, "name");
        Guard.ForNull(address, "address");
        Guard.ForDateInTheFuture(birthdate, "birthdate");
        Id = id;
        Name = name;
        Address = address;
        Balance = 0;
    }
    public Guid Id { get; private set; }
    
    public Name Name { get; private set; }
    public Address Address { get; private set; }
    public DateTime BirthDate { get; private set; }
    private decimal Balance { get; set; }
    private bool HasSufficientFunds(decimal value) 
    {
        return Balance - value >= 0;
    }
    public void DecreaseBalance(decimal value)
    {
        if (HasSufficientFunds(value))
            throw new InsufficientFundsException();
        Balance -= value;
    }
    public void MoveTo(Address newAddress)
    {
        Guard.ForNull(newAddress, "new address");
        Address = newAddress;
    }
}

Summarized

By making my entities immutable from the start I am guided into adhering to a bunch of good practices:

A constructor
Comparing identities
Defaults
Protecting invariants
Composition and extracting value objects
Tell, don't ask
Making operations explicit
Opening up bit by bit

I hope this post made some sense. I'd love to hear from you how I could improve this process, and which process you have accustomed yourself with over the years.

Reading large files in chunks with proper encapsulation

2013-03-24T18:16:00.000+01:00

I've been doing some work lately which involves sequentially reading large files (> 2 to 5GB). This entails that it's not an option to read the whole structure in memory; it's more reliable to process the file in chunks. I occasionally come across legacy that solves exactly this problem, but in a procedural way, resulting in tangled spaghetti. To be honest, the first piece of software I ever wrote in a professional setting also went at it in the wrong way.

There is no reason to let it come to this though; you can use the often overlooked yield return keyword to improve encapsulation.

When you use the yield keyword in a statement, you indicate that the method, operator, or get accessor in which it appears is an iterator. You consume an iterator method by using a foreach statement or LINQ query. Each iteration of the foreach loop calls the iterator method. When a yield return statement is reached in the iterator method, expression is returned, and the current location in code is retained. Execution is restarted from that location the next time that the iterator function is called.

Have a look at the following Reader class which takes advantage of yield returning. This class reads from file, line by line, building a chunk, to return it when the desired chunk size is attained. In the next iteration, the call will continue by clearing the lines - thereby releasing memory, and rebuilding the next chunk.

public class Reader
{
    private int _chunkSize;
    public Reader(int chunkSize) 
    {
        _chunkSize = chunkSize;
    }
    public IEnumerable<Chunk> Read(string path)
    {
        if (string.IsNullOrEmpty(path))
            throw new NullReferenceException("path");
        var lines = new List<string>();
        using (var reader = new StreamReader(path))
        {
            string line;
            while ((line = reader.ReadLine()) != null)
            {
                lines.Add(line);
                if (lines.Count == _chunkSize)
                {
                    yield return new Chunk(lines);
                    lines.Clear();
                }
            }                
        }
        yield return new Chunk(lines);
    }
}
public class Chunk
{
    public Chunk(List<string> lines) 
    {
        Lines = lines;
    }
    public List<string> Lines { get; private set; }
}

And that's one way to achieve clean encapsulation without starving your machine's memory.

var reader = new Reader(chunkSize: 1000);
var chunks = reader.Read(@"C:\big_file.txt");
foreach (var chunk in chunks)            
    Console.WriteLine(chunk.Lines.Count);

Putting my IronMQ experiment under stress

2013-03-17T16:10:00.000+01:00

Two weeks ago, I shared my first impressions of IronMQ. Last week, I looked at some infrastructure to facilitate pulling from IronMQ. This implementation worked, but I hadn't put it under stress yet; "First make it work, then make it fast", and all of that.

I arranged a simple scenario for testing: one message type - thus one queue, where there are eight queue consumers that simultaneously pull messages from that queue, and dispatch them to a handler which sleeps for one second.

public class MessageSleepForOneSecond { }
public class MessageSleepForOneSecondHandler : IMessageHandler<MessageSleepForOneSecond>
{
    public void Handle(MessageSleepForOneSecond message)
    {
        Thread.Sleep(1000);
    }
}

To establish a baseline, I foolishly set the polling interval to only 100ms, and pulled 2000 messages from the queue one at a time. With this configuration I processed all 2000 messages in 2 minutes and 20 seconds, with an average throughput of 14.3 messages per second. In theory you would expect the throughput to be higher though.
The constraint in this story is the CLR's thread pool. Every time a queue consumer's internal timer ticks, the callback which pulls from the queue and invokes the messagehandler, takes up a new thread on the thread pool. The thread pool makes a few threads available when you start your application, but once they're all in use, it will have to start new ones, which is rather expensive. More importantly though, when you're queuing too many tasks on the thread pool, and the number of active threads is higher than the number of processors, it will slow down, and wait 500ms to see if it can reuse the existing threads, before creating a new one. When the maximum number of threads is reached, the thread pool will still enlist your tasks in its queue, but only start processing them once threads become available again. In short, the thread pool has a few tricks up its sleeve to protect you from saturating your resources. Remember that too much parallelization and its corresponding context switches won't do you any good.

Having established a baseline, and having learned a bit more on how the thread pool behaves, I tried one of the first optimizations I already had in mind last week; pulling batches instead of single messages. This reduces the number of necessary HTTP requests, and the number of threads needed to do work on. To support this, I extended the queue consumer configuration with a new property, and changed the queue consumer to take the batch size into account.

public interface IQueueConsumerConfiguration<T>
{
    int PollingInterval { get; }
    int BatchSize { get; }
}
try
{
    var messages = (IEnumerable<Message>)null;
    if (!_queue.TryGet(out messages, _queueConsumerConfiguration.BatchSize))
        return;
    foreach (var message in messages)
    {
        try
        {
            var messageBody = (T)JsonConvert.DeserializeObject(message.Body, typeof(T));
            _messageDispatcher.Dispatch<T>(messageBody);
            _queue.Delete(message.Id);
        }
        catch (Exception ex)
        {
            _errorHandler.Handle(ex, message);
        }
    }
}
catch (Exception ex)
{
    _errorHandler.Handle(ex, null);
}

On repeating the test with 2000 messages, the same polling interval of 100ms, but with a batch size of 30, the messages were now all processed in one minute and fifteen seconds, resulting in a throughput of 26 messages per second. That's almost an improvement of 100%.

This throughput isn't sustainable though if we had a lot more messages to process. We're starting a new thread every 100ms or 500ms, while the work we are doing on it only finishes after a rough 30 seconds (it's not only invoking the handlers, but the HTTP requests also take time). We're burning through threads quicker than we're releasing them. If we would run out of threads on the thread pool, it would just stop starting new ones, and queue the tasks until other threads are done doing work.

In my previous post I also considered a smart polling algorithm, but I haven't looked at that yet, what's in place is more than good enough for me at the moment.

Be sure to take these numbers with a grain of salt. I would have to test my infrastructure with millions of messages on the queue instead of just 2000 to get trustworthy results. I feel I can predict fairly well how the system will behave when put under load for a longer amount of time though; it would grind to a halt. As mentioned before, we would run out of threads to do work on. I simulated this by lowering the thread pool's maximum number of threads. Other parameters that influence the numbers in this test are: size of the messages, version of the runtime, the operating system, the amount of processors, latency of the network... I ran these tests with empty messages, .NET 4 installed on my own Windows 7 box with an Intel i7 on board.

It comes down to cherry picking a configuration per queue consumer that will be sustainable based on the amount of messages you expect, the desired throughput, and the time it takes to process a single message.

Find the source on Github.

Some experimental infrastructure for IronMQ pull

2013-03-10T17:29:00.000+01:00

I wrote about using IronMQ as a cloud-based message queue last week. In that post I explained that you can go at using IronMQ in two ways; either you pull from the queue yourself, or you let IronMQ push messages from the queue to your HTTP endpoints. At first sight, the latter allows you to outsource more infrastructure to their side, but upon closer inspection it also introduces other concerns: security, local debugging and scalability.

Out-of-the-box there is no infrastructure in the client libraries to facilitate periodic pull - polling, that's why I took a stab at doing it myself. It's still rough, not production tested, and hasn't considered a bunch of niche scenarios, but it should give you an idea of the direction it's going.

A high-level overview of the components looks like this.

A queue host hosts multiple queue consumers. Each queue consumer will poll a queue for one type of message on a configurable interval, and dispatch dequeued messages to relevant message handlers. After handling the message, the message will be deleted from the queue. If something happened to go wrong, the error handler for this type of message will be invoked, and the message will automatically return to the queue.

I'm going to look at each component, starting with the smallest, and slowly assemble them into bigger components.

Each queue consumer can be configured independently. For now, only the polling interval can be changed. By default it's one second, or 1000 milliseconds.

public interface IQueueConsumerConfiguration<T>
{
    int PollingInterval { get; }
}
public class QueueConsumerConfiguration<T> : IQueueConsumerConfiguration<T>
{     
    public int PollingInterval
    {
        get { return 1000; }
    }       
}

A queue can push messages, get raw messages, and delete them. The implementation makes use of the OSS IronIO client libraries.

public interface IQueue<T>
{
    bool TryGet(out Message message);
    void Delete(string messageId);
    void Push(T message);
}
public class Queue<T> : IQueue<T>
{
    private readonly IronIO.IronMQ _queue;
    public Queue(string projectId, string token)
    {
        Guard.ForEmptyString(projectId, "projectId");
        Guard.ForEmptyString(token, "token");
        var queueName = typeof(T).Name; 
        _queue = new IronMQ(queueName, projectId, token);
    }
    public bool TryGet(out Message message) 
    {
        message = _queue.Get();            
        return message != null;
    }
    public void Delete(string messageId)
    {
        Guard.ForNull(messageId, "messageId");
        _queue.Delete(messageId);
    }
    
    public void Push(T message)
    {
        Guard.ForNull(message, "message");
        _queue.Push(JsonConvert.SerializeObject(message));
    }
}

A message dispatcher dispatches messages to the relevant handlers.

public interface IMessageDispatcher
{
    void Dispatch<T>(T message);
}
public class MessageDispatcher : IMessageDispatcher
{
    private readonly IKernel _kernel;
    public MessageDispatcher(IKernel kernel)
    {
        _kernel = kernel;            
    }
    public void Dispatch<T>(T message)
    {
        var handlers = _kernel.GetAll<IMessageHandler<T>>();
      
        foreach (var handler in handlers)
            handler.Handle(message);                        
    }
}

If something goes wrong pulling the message from the queue or handling it, the error handler will be invoked passing in the exception and the raw message. Since it's possible that something is wrong with the message in itself, I pass in the raw message with the serialized message and all its meta data like id, delay and expiration date.

public interface IErrorHandler<T>
{
    void Handle(Exception exception, Message message);
}
public class ErrorHandler<T> : IErrorHandler<T>
{       
    public void Handle(Exception exception, Message message)
    {
        throw exception;
    }     
}

Putting all these components to work together, we end up with a queue consumer. When a queue consumer is started, it instantiates a timer which will try to get a raw message from the queue on each tick. If there's a raw message, it will extract the body, deserialize it into the message, dispatch it, and finally delete the raw message from the queue. If something goes wrong here, the error handler will be invoked, and the message will automatically return back to the queue.

public interface IQueueConsumer<T> : IQueueConsumer where T : class
{
}
public interface IQueueConsumer : IDisposable
{
    void Start();
}
public class QueueConsumer<T> : IQueueConsumer<T> where T : class
{
    private readonly IQueue<T> _queue;
    private readonly IErrorHandler<T> _errorHandler;
    private readonly IQueueConsumerConfiguration<T> _queueConsumerConfiguration;
    private readonly IMessageDispatcher _messageDispatcher;
    private Timer _timer;        
    public QueueConsumer(
        IQueue<T> queue, 
        IErrorHandler<T> errorHandler,
        IQueueConsumerConfiguration<T> queueConsumerConfiguration,
        IMessageDispatcher messageDispatcher)
    {
        _queue = queue;
        _errorHandler = errorHandler;
        _queueConsumerConfiguration = queueConsumerConfiguration;
        _messageDispatcher = messageDispatcher;
    }    
    public void Start()
    {
        _timer = new Timer((x) =>
        {
            var message = (Message)null;
            var messageBody = (T)null;
            try
            {
                if (!_queue.TryGet(out message))
                    return;
                messageBody = (T)JsonConvert.DeserializeObject(
                    message.Body, typeof(T));
                _messageDispatcher.Dispatch<T>(messageBody);
                _queue.Delete(message.Id);
            }
            catch (Exception ex)
            {
                _errorHandler.Handle(ex, message);
            }                
        }, null, 0, _queueConsumerConfiguration.PollingInterval);            
    }
    public void Dispose()
    {
        if (_timer == null)
            return;
        _timer.Dispose();
    }
}

Since we have multiple queues to pull from, we can use a queue host to control multiple queue consumers at once. The queue host configuration decides which queue consumer to instantiate and start.

public class QueueHostConfiguration
{
    public QueueHostConfiguration(IEnumerable<Type> messageTypes)
    {
        Guard.ForNull(messageTypes, "messageTypes");
        MessageTypes = messageTypes;
    }
    public IEnumerable<Type> MessageTypes { get; private set; }
}
public class QueueHost : IDisposable
{
    private readonly IKernel _kernel;
    private readonly QueueHostConfiguration _configuration;
    private readonly List<IQueueConsumer> _consumers;
    public QueueHost(IKernel kernel, QueueHostConfiguration configuration)
    {
        _kernel = kernel;
        _configuration = configuration;
        _consumers = new List<IQueueConsumer>();
    }
    public void Start()
    {
        foreach (var messageType in _configuration.MessageTypes)
        {
            var queueConsumerType = typeof(IQueueConsumer<>).MakeGenericType(messageType);
            var queueConsumer = (IQueueConsumer)_kernel.Get(queueConsumerType);
            _consumers.Add(queueConsumer);
            queueConsumer.Start();
        }
    }
    public void Dispose()
    {
        if (_consumers == null)
            return;
        foreach (var consumer in _consumers)
            consumer.Dispose();
    }
}

In your application, you'll end up doing something like this to start the queue host.

using (var host = new QueueHost(kernel, new QueueHostConfiguration(
    new[] { typeof(MyMessage) })))
{
    host.Start();
    Console.ReadLine();
}

All the components are glued together using Ninject and some conventions.

public class Bootstrapper
{
    public void Run(IKernel kernel)
    {
        kernel.Bind(x => x
          .FromThisAssembly()
          .SelectAllClasses()
          .InheritedFrom(typeof(IMessageHandler<>))
          .BindAllInterfaces());
        kernel.Bind<Infrastructure.IMessageDispatcher>()
            .To<Infrastructure.MessageDispatcher>()
            .InTransientScope();
        kernel.Bind(x => x
            .FromAssemblyContaining(typeof(IQueue<>))
            .SelectAllClasses()
            .InheritedFrom(typeof(IQueue<>))
            .BindAllInterfaces()
            .Configure(y =>
            {
                y.WithConstructorArgument("projectId", z => { return "your_project_id"; });
                y.WithConstructorArgument("token", z => { return "your_token"; });
            }));
        kernel
            .Bind(typeof(Infrastructure.IQueueConsumer<>))
            .To(typeof(Infrastructure.QueueConsumer<>));
        kernel
            .Bind(typeof(Infrastructure.IQueueConsumerConfiguration<>))
            .To(typeof(Infrastructure.QueueConsumerConfiguration<>));
        kernel
            .Bind(typeof(Infrastructure.IErrorHandler<>))
            .To(typeof(MyErrorHandler<>));    
    }
}

This is what I got for now. Next step is to do some more serious integration testing, and see what gives. There are two things I already kind of expect to run into; the maximum number of concurrent connections, and thread starvation (each timer tick starts a new thread). Anything else I'm going to run into?

The biggest disadvantage of opting for pull that is already obvious now, is the possible number of wasted HTTP requests. You could increase the polling interval, and thereby lower the number of requests, but this would harm the throughput of message bursts. Something I'm considering right now, is introducing a smart polling algorithm. Another option that will lower the number of requests, is to pull batches instead of single messages from the queue. Implementing this one will be rather straightforward, yet improve things considerably.

First IronMQ impressions

2013-03-03T16:45:00.000+01:00

First time I touched messaging was in the first few years of my professional life working on software that supported fire departments in their day-to-day activities. The dispatching software would send messages to a proprietary broker, which in its turn would forward them to interested subscribers; other dispatching clients, or services. To ensure availability, the broker component could failover to a different machine, but that was about it. It didn't allow you to queue or retry messages; if you weren't up when the messages were forwarded, you would never receive them. When the brokers were both down, all messages would be lost; the clients didn't have infrastructure out-of-the-box that could queue the messages locally until it came back up again. When things went haywire, and they occasionally did, these missing features would often leave us with an inconsistent state. More modern messaging software has solved these concerns though.

Although quite a few systems would benefit from asynchronous and loosely coupled messaging - especially to improve reliability and (perceived) performance, but also scalability, I still too seldom see or hear about projects that get to go that extra mile. Solutions often end up compromising in quality to avoid introducing that extra component and those unconventional questions. And this decision might be perfectly sound, because lots of factors are at play, not just technical ones. It's still a pity when you see solutions struggle to solve a problem in a decent way because they're stuck with synchronous communication.

Imagine a public website that's in the business of booking hotels. The offers they show to their customers are all based on data provided by third parties. Because it's so expensive to fetch this data, it's being cached, and as a result, it's stale seconds after fetching it. The moment a user confirms, they could fetch fresh data from the relevant third party to make sure the room is still available, but this process is error prone: the third party might be down, fetching the data is really slow, one of our their own components might be down, on conflicts they might want to compile a list of some decent alternatives, which in its turn might also be too slow if done on demand. One alternative could be to queue the booking, and process it in the back-end. Once they're done processing it, they can mail the user a confirmation or an apology with a list of alternatives attached. By making this process asynchronous, they avoid the risk of a slow user experience and clumsy failures they can't recover from, making them lose business in the long run. But then again, they also take the burden of extra infrastructure, new operational concerns and different questions. Trade-offs.

Anyways, in this blog post, I wanted to share some first impressions on IronMQ. I admittedly kind of accidentally discovered this service browsing through AppHarbor's add-ons. Here it is described as "a scalable cloud-based message queue".

Basically IronMQ gives you a REST enabled queue in the cloud. After authenticating, you can POST a new message to the queue. If you are unable to POST the message, you'll lose it, since there is no support out-of-the-box for retrying or persisting the message on the client. Once the message is posted, another process can GET the message, and DELETE it after successfully processing it. If something happened to go wrong while processing the message, the message will return to the queue, and be retried one minute later. If the message processing keeps failing on subsequent retries, the retries won't repeat infinitely though; messages expire (the default is 7 days, and the maximum is 30 days). This is exactly the kind of infrastructure we need to support the asynchronous booking scenario: have the customer put its booking on the queue, and one of our background processes will try to process it; if something goes wrong, we'll just keep retrying for a while.

The REST API is simple, yet there are client libraries available for most popular languages. They don't provide that much extra functionality though. Here's the gist.

var queue = new IronMQ(queueName, projectId, token);
queue.Push("{ hello: world }");
var message = queue.Get();
queue.Delete(message.Id);

In this scenario you're responsible for pulling data from the queue. This is just one way to go at things though; another option is to let IronMQ push messages to your HTTP endpoints. While this allows you to outsource some infrastructure to their side, it raises other concerns:

Security: you might need to enable HTTPS, and provide an authentication mechanism.
Debugging: if you want to do some end-to-end integration testing on your local machine, you'll need to give your machine a public IP and set up something like dyndns.
Scalability: depending on the expected message volume, and the web stack you're rolling with, it might be more expensive to have to set up all these web servers, instead of a few background workers.

Errors are handled quite elegantly; once you processed the message successfully, make your endpoint return HTTP status code 200 or 202, and the message will be removed from the queue. HTTP status code 202 is used for long running processes. If the response code is in the 400 or 500 range, the message will return to the queue to be retried later.

When the expected volume of messages is rather small, it makes more sense to opt for push; you don't waste that many HTTP requests.

IronMQ makes it extremely simple to get started; go to their site, get a project id and a token, and start making HTTP calls. Do them yourself, or use one of the client API's. But it also seems to be all you're going to get; there is no infrastructure that addresses operational concerns, error queues, retry strategies, local queues,... IronMQ provides you with raw queueing infrastructure, not a framework.

Their site does give you a look into your queues though; you can't look at the messages, but you do get a nice overview.

I don't mind that it's not a turnkey solution though; I learn a lot from tinkering with this stuff. Solving problems and considering trade-offs for yourself is priceless.

Using HTTP for messaging still feels a bit quirky to me. As long as I can remember people have been making me believe HTTP is not the best fit for high-throughput messaging scenarios, and I do understand their motivations somewhat. But when even databases start to embrace HTTP, it's probably time to shake off the doctrine. It's so comfy to not have to understand a new protocol, and HTTP just seems such a sensible thing to do when you're off-premise.

I'm going to conduct some more experiments this week, I'll see what gives.

My Christmas holiday project postmortem

2013-02-24T16:42:00.001+01:00

Somewhere over a year and a half ago I discovered the music of Dire Straits, which has sparked a fanatical love and fascination for the guitar in me, and basically for every piece of music Mark Knopfler has ever touched (*). A year ago, I finally had the courage to pick up the guitar myself. Not sure if I'd stick with it, I made an uninformed purchase of a rather inexpensive Squier Jazzmaster, just because it somewhat resembled the real object of desire, a Fender Stratocaster. Three months ago, I got rid of the Jazzmaster, and bought myself a tango red Mexican Stratocaster, and I'm in love with it. Yet, I have also become very fond of the sound of a Les Paul these days. Having just bought the Strat, I thought I could maybe find a cheap used Les Paul online.

I set out to find one on the most popular Belgian online secondhand marketplace (600k visitors daily), so I started browsing their listings daily. This turned out to be rather cumbersome and inefficient: constantly repeating the same process, items were already sold before I could make a bid, there were no new items since the last time I visited... It didn't take me long before I started thinking of a way to automate this dull process. Looking at other marketplaces, I found that some unburden their users by providing notifications; push instead of pull. Maybe I could build something similar?

Doing a bit of research over the weekend, I found out that they expose, what I call, an accidental API; written in first place to provide a snappy mobile user experience, not to expose all their data to third parties. Having an uncomplicated way of searching their data, I built a bit of code on top of it. Nothing all too fancy though; a Quartz job which periodically queries their service, parses the results and stores them in a RavenDB database. A few times a day, these search results are then compiled, and sent to my inbox using MailGun. All of this running for free on AppHarbor.

Showing all of this to the girlfriend, she asked if I could set up the same thing for her, but then for a specific type of camera. This is when I decided it could be useful enough to make it publicly available. I thought of a few ways to support the costs of hosting (which are extremely low): embed ridiculously relevant ads in the mails, make people pay for more frequent polling, or sell it (although very unlikely).

Two weeks later, I had something working online.

But how do I inform people of its existence? I first thought of using Twitter to monitor for relevant tweets where people ask for something secondhand. I started out by using a new dedicated generic account, but this got suspended rather quickly. In the second iteration, I used my personal account. This yielded better results; people saw I was human, and regularly thanked me for the tip. They didn't sign up too often though..

Not willing to give up on the idea already, I used Google Adwords to advertise on the online marketplace directly. The results of this campaign were sobering, but extremely valuable; people just didn't care. As a side note; secondhand seems to be quite an expensive keyword!

In hindsight, I can think of plenty of reasons why this somewhat useful project has no success:

The problem it is trying to solve obviously isn't enough of a pain!
People don't Google for it, and even if they did, this site had hardly any chance in making it to the first page.
In general the secondhand offering is enormous, and people often quickly settle for less.
Specialized markets aren't situated on these sites.
People don't like giving away their email address, definitely not to strangers they don't trust.
Everyone struggles to keep their inboxes clean, receiving mail puts off people.

Damn, it's always evident in hindsight.

Here are a few things I learned/got confirmed:

Being your own customer is priceless; you know exactly where the value is at.
Chances are you will never see a user; don't spend too much time optimizing for scalability and reliability.
Some Google ads plus a website with a simple form can be enough to let you cheaply validate the worthiness of pursuing an idea.

(*) Take some time to explore his work: brothers in arms, postcards from Paraguay, song for Sonny Liston, you and your friend, and so many more are worth a listen.

Adding the R to CQS: some storage options

2013-02-17T17:57:00.000+01:00

I've been writing quite a bit about CQS (or command and query separation) lately. In my last post on using events, I already hinted towards bringing in the R; command and query responsibility separation.

With CQS, commands can mutate data, while queries can only read that data. CQRS takes this one step further, and assigns commands and queries each a dedicated model; we now talk of a write side, and a read side.

I like Clemens Vasters definition best.

CQRS is a simple pattern that strictly segregates the responsibility of handling command input into an autonomous system from the responsibility of handling side-effect-free query/read access on the same system. Consequently, the decoupling allows for any number of homogeneous or heterogeneous query/read modules to be paired with a command processor. This principle presents a very suitable foundation for event sourcing, eventual-consistency state replication/fan-out and, thus, high-scale read access. In simple terms, you don’t service queries via the same module of a service that you process commands through. In REST terminology, GET requests wire up to a different thing from what PUT, POST, and DELETE requests wire up to.

A nice drawing also helps in understanding CQRS (from the CQRS journey material).

Although scalability seems to be one of the big selling points of CQRS, there are still some valid arguments applicable to my world; the strongest one being able to avoid the discrepancy which exists while you use the same model for reading and writing. I think everyone suffers from this one regularly. A popular and realistic example is this one; an ORM is used to map our domain model to a relational database, the tables are mapped very closely to the structure of the domain model. Not long after, it becomes evident that it's impossible to write simple and performant queries targeting this datastructure. We could optimize for reads, but this would impact the complexity and performance of writing. With CQRS, reads and writes are segregated; we can now optimize both parts independently. And this doesn't only result in being able to show a list faster on a user's screen, but interesting things can also be done to empower reporting and data mining; think of how often using the same database for these tasks makes it hard and expensive to change things.

While I still have done very little with CQRS, I have been looking at more and more real world examples, trying to fill in the blanks. What always has been kind of vague to me, is how you go at storing your domain model, and your read models in practice. Here are a few possible techniques - these are some proven techniques, and partially my own presumptions (you hardly find any OSS brown field examples).

The compromise

CQRS doesn't have to be an application-wide architecture necessarily; nothing stops you from introducing it gently, and just applying it to parts of your application where the added value is over-obvious. This could mean that you use a conventional architecture; a relational database with an ORM, or a document store, not distinguishing the write side from the read side. Yet for certain scenarios, you could introduce a specialized read or write side. For example; update the statistics read model on every relevant write, update a denormalized optimized read model for searches, etc..

NORM

While the relational paradigm definitely has its place, mapping your domain to the database can get complex, and require much maintenance. If you don't expect of your write side to be queryable, you can take advantage of less cumbersome techniques such as a key value store to store your domain model. This does force you to completely separate reads from writes though.

Event Sourcing

When you look at most OSS CQRS implementations, Event Sourcing and CQRS go hand in hand. With Event Sourcing, you capture all application state changes as a sequence of events. I'm really fond of the theory behind this pattern, and I can imagine the added operational value of having a log of each change. Yet, I also think you could largely achieve the same result by enabling journaling and adding some interception. Storage wise, you store all the event streams in an event store, which is optimized for such a task. Your read side can again be whatever you fancy.

These three techniques aren't mutually exclusive. There are a bunch of arguments to consider, and everything is highly dependent on your technical and operational requirements.

What is your experience with CQRS? Which techniques have you applied in practice?

Premature judgment

2013-02-10T16:46:00.001+01:00

When I started my first job, I hardly ever judged my peers. After all, how could I? Everything was unknown for me; I couldn't differentiate good from bad. Over the years that has changed a bit, but with that, I've also slowly become more judgmental towards peers, often prematurely, and not always deservedly.

The first few months of last year, I found myself doing maintenance on a legacy code base between projects. While I worked my way through layer after layer, I pointed my frustration towards those that had come before me; they were responsible for putting me in this mess. With half the office having touched the code base, that didn't really add up though. When I looked at the commit history of some of the offending modules, I found names that I didn't expect; those people were still around, and I actually thought of them pretty highly.

Judging someone's competence solely by code he has written in the past is a flaw. There are very little pieces of code I have written over the years where I still feel comfortable about today. When I reflect on what made it go wrong, I don't have to look far to find a bunch of reasons to blame it on; consistency was favored over common sense, major breakthroughs occurred only after the project was already in maintenance mode, people inexperienced with the domain and infrastructure were dumped on the project last minute to make up for bad planning, knowledge of the technology stack hadn't matured, some patterns and practices weren't commonplace yet, etc... I always find plenty of reasons to shift blame, but when I look at code written by someone else, it has to be their own fault; they must not be very good at building software. And this is unfair; I have no way of knowing the constraints they had to deal with, nor the context they had to work in. None of these justify neglecting basic hygiene though!

I tried to come up with other things that influence my opinion on someone before having actually worked with them; I found two.
The biggest influencer is word of mouth. I try to surround myself with people that share a similar thinking, and if that trusted circle has a strong opinion on someone else, I take note.
The last influencer is someone's online presence. When I learn of someone new joining ranks, I can't resist to look up what he's doing online. Twitter, Facebook or a blog can give away quite a bit.

All in all, I think some preconception is human, and might be the result of subconsciously protecting your work. You only want to involve those with whom you will enjoy working towards your shared goal.

Have you experienced similar behavior? When do you judge prematurely?

Raising events in commandhandlers

2013-02-03T17:49:00.000+01:00

I've explored quite a few options on how to handle commands and queries in the last few posts. I finally settled on this approach. The example used in that post looked like this.

public class CreateSubscriptionCommandHandler : ICommandHandler<CreateSubscriptionCommand>
{    
    private IDocumentSession _session;
    public CreateSubscriptionCommandHandler(IDocumentSession session)
    {
        _session = session;
    }
    public void Handle(CreateSubscriptionCommand command)
    {
        var subscription = new Documents.Subscription(
            command.Value, command.Category, command.EmailAddress);
        _session.Store(subscription);    
    }
}

Now imagine I would want to do some extra stuff after creating the subscription; update the sales statistics, append the email address to a mailing list, send out a confirmation email, etc..

You could go at this by simply extending the commandhandler, but the problem here is that you quickly end up with a bulky and dependency-heavy commandhandler, which will quickly fail to communicate its intent.

One solution could be to introduce events to decouple things in smaller pieces, and to help communicate intent more clearly.

The infrastructure to handle events is rather straightforward, and can be based on Udi Dahan's Domain Events Salvation.

public class Events : IEvents
{
    private readonly IKernel _kernel;
    public Events(IKernel kernel)
    {
        _kernel = kernel;
    }
    public void Raise<T>(T @event) where T : IEvent
    {
        var handlers = _kernel.GetAll<IEventHandler<T>>();
        foreach (var handler in handlers)        
            handler.Handle(@event);        
    }     
}

When an event is raised, the eventing infrastructure will look in the container for implementations that can handle the event, and invoke them in a random order.

Raising an event from the commandhandler can be done by injecting this extra piece of infrastructure.

public class CreateSubscriptionCommandHandler : ICommandHandler<CreateSubscriptionCommand>
{    
    private IDocumentSession _session;
    private readonly IEvents _events;
    public CreateSubscriptionCommandHandler(IDocumentSession session, IEvent events)
    {
        _session = session;
        _events = events;
    }
    public void Handle(CreateSubscriptionCommand command)
    {
        var subscription = new Documents.Subscription(
            command.Value, command.Category, command.EmailAddress);
        _session.Store(subscription);    
        
        _events.Raise(new SubscriptionCreatedEvent(query.Id));
    }
}

The SubscriptionCreatedEvent class is a simple value object, which exposes the subscription identifier.

public class SubscriptionCreatedEvent : IEvent
{
    public SubscriptionCreatedEvent(string subscriptionId)
    {
        SubscriptionId = subscriptionId;
    }
    public string SubscriptionId { get; private set; }
    public override bool Equals(Object other)
    {
        if (other == null)
            return false;
        var otherEvent = other as SubscriptionCreatedEvent;
        if (otherEvent == null)
            return false;
        return otherEvent.SubscriptionId == SubscriptionId;
    }
    public override int GetHashCode()
    {
        return SubscriptionId.GetHashCode();
    }
}

To subscribe to this event, implement the IEventHandler interface, and register the implemenation in the container.

public interface IEventHandler<T> where T : IEvent
{
    void Handle(T @event);
}
public class SendConfirmationMailOnSubscriptionCreated : IEventHandler<SubscriptionCreatedEvent>
{    
    public void  Handle(SubscriptionCreatedEvent @event)
    {
        ...
    }
}
public class UpdateSalesStatisticsOnSubscriptionCreated : IEventHandler<SubscriptionCreatedEvent>
{    
    public void  Handle(SubscriptionCreatedEvent @event)
    {
        ...
    }
}

Eventhandlers are invoked synchronously, and participate in the commandhandler's unit of work, so if something goes haywire in one of the eventhandlers, nothing gets committed, not even what happened in the original commandhandler. Depending on your requirements, you might want to handle this differently though.

With this approach, tests also become more compact. Commandhandler tests now only need to assert that the event gets raised, and all the other logic gets offloaded to separate tests per eventhandler.

Summary

By introducing events, you can decouple commandhandlers into more focused, and intent-revealing bits. Your tests are the perfect proof of how much cleaner things get. One of the cues to listen for is when you do x or on doing y, also do z.

Are you using events? If so, domain events, or its big brother Event Sourcing?

Organizing commands and queries

2013-01-27T18:23:00.000+01:00

In the last few posts I settled on an architecture for handling commands and queries. A byproduct of the described approach, is that your codebase quickly racks up plentiful little classes; a class to hold data, and a handler to act on that data, for each use case.

There are a few ways you can go at organizing things.

Everything in one location

When there is very little going on in your application, you can just dump everything in one location without getting hurt too much.

Folder per functionality

When your application grows, and the coherency between different use cases are obvious, you can just use folders - and corresponding namespaces, to organize your commands and queries, and to draw their functional boundaries. Uncle Bob and Mark Needham have sold me on structuring my code based on functionality instead of technical concepts.

Keeping each commandhandler in a separate class is especially interesting when they are rather bulky and contain a good amount of logic. You can think of each class as a little piece of functionality in itself. Take a look at the RavenDB codebase to get an idea of what that could look like.

It also feels like this way tends to bring your code closer to the Solution Explorer; just one double-click and you are looking at your implementation; no scrolling or searching between method definitions necessary. Maybe the problem is now just shifted a level higher in the hierarchy though.

Composing a service class

You can also opt to group commandhandler implementations in one service class. This variation might make more sense when your implementations are rather skinny, and don't do a whole lot but translating and forwarding your invocation.

public class SubscriptionService : 
        ICommandHandler<SubscribeCommand>,
        ICommandHandler<UnsubscribeCommand>
{
    public void Handle(SubscibeCommand command)
    {
        throw new NotImplementedException();
    }
    public void Handle(UnsubscribeCommand command)
    {
        throw new NotImplementedException();
    }
}

Use the hints your dependency graph gives you to find a composition that makes sense.

How do you go at organizing commands and queries?

RavenDB: Drop all collections

2013-01-24T20:08:00.000+01:00

I never stub or mock the database when I'm using RavenDB. Generally, I use an embeddable documentstore running in memory, and initialize a new instance on every test. However, I like to run some stress tests against a real instance, and here I found myself wanting to wipe clean the state of previous tests, without having to create a new database (which is rather slow).

First I create the default DocumentsByEntityName index to make sure it's there - it normally gets created when you open the studio for the first time. Then I use one of the advanced database commands: DeleteByIndex, and query all the tags.

using (var session = _documentStore.OpenSession())
{
    new RavenDocumentsByEntityName().Execute(_documentStore);
    session.Advanced.DatabaseCommands.DeleteByIndex(
            "Raven/DocumentsByEntityName",
            new IndexQuery { Query = "Tag: *" });                
}

This technique doesn't seem to be widely used judging by the first page of Google search results. If there is a reason for that though, let me know!

Separating command data from logic and sending it on a bus

2013-01-20T22:03:00.002+01:00

In my first post on this topic, I started out with an attempt to limit abstractions to solely commands and queries. Commands and queries were self-contained and could be invoked by passing them to a context-providing generic handler. The drawback of this approach was that it made constructor dependency injection impossible. In a next post, I separated data from logic, but never got around to writing a dispatcher that associates command data with their handlers. Last week, I revisited the first approach, and added an unconventional implementation of injecting dependencies by an Inject method convention.

I still believe that last approach is very simple and works fine if extra dependencies are exceptional. I do admit that it will make architectural shoots harder to handle; everything is rather tightly coupled. So let's look at an alternative architecture which pulls all the bits apart, and should be better equipped to handle change.

Let's first separate command data from logic.

public class CreateSubscriptionCommand 
{     
    public CreateSubscriptionCommand(string value, string category, string emailAddress)
    {
        Guard.StringIsNullOrEmpty(value, "value");
        Guard.StringIsNullOrEmpty(category, "category");
        Guard.StringIsNullOrEmpty(emailAddress, "emailAddress");
        Value = value;
        Category = category;
        EmailAddress = emailAddress;
    }
    public string Value { get; private set; }
    public string Category { get; private set; }
    public string EmailAddress { get; private set; }
    public override bool Equals(Object other)
    {
        if (other == null)
            return false;
        var otherCommand = other as CreateSubscriptionCommand;
        if (otherCommand == null)
            return false;
        return otherCommand.Value == Value && 
            otherCommand.Category == Category && 
            otherCommand.EmailAddress == EmailAddress;
    }    
    public override int GetHashCode()
    {
        return Value.GetHashCode() ^ 
            Category.GetHashCode() ^ 
            EmailAddress.GetHashCode();
    }
}

The data is just a POCO. Notice the equality overrides; this comes in handy when you're testing.

The class that handles on the data needs to implement the ICommandHandler interface.

public class CreateSubscriptionCommandHandler 
    : ICommandHandler<CreateSubscriptionCommand>
{    
    private IDocumentSession _session;
    public CreateSubscriptionCommandHandler(IDocumentSession session)
    {
        _session = session;
    }
    public void Handle(CreateSubscriptionCommand command)
    {
        var subscription = new Documents.Subscription(
            command.Value, command.Category, command.EmailAddress);
        _session.Store(subscription);    
    }
}

Compared to the previous approach, we're now injecting the session instead of having it handy as a property; that coupling is now completely gone.

Last thing left to do is create an interface that consumers can use to send commands on: a bus.

public class Bus : IBus
{
    private readonly IKernel _kernel;
    public Bus(IKernel kernel)
    {
        _kernel = kernel;
    }
    public void ExecuteCommand<T>(T command) where T : class
    {
        var handler = _kernel.Get<ICommandHandler<T>>();
        handler.Handle(command);
        _kernel.Get<IDocumentSession>().SaveChanges();
    }
}

The ExecuteCommand method dispatches data to the correct handler by resolving it from the container, and also commits the unit of work.

The consumer can execute commands like this.

bus.ExecuteCommand(
    new CreateQueryCommand(queryValue, category, emailAddress));

With this approach having all the bits spread, we have a bit more work gluing all the pieces together. The session is now known in the container, and is request scoped. The commandhandlers are also all registered in the container.

protected override void ConfigureRequestContainer(IKernel container, NancyContext context)
{        
        // you don't want to register them all individually
        container
            .Bind<ICommandHandler<CreateSubscriptionCommand>>()
            .To<CreateSubscriptionCommandHandler>();        
        // snip..
        container.Bind<IDocumentSession>()
            .ToMethod((ctx) => { 
                return ctx.Kernel.Get<IDocumentStore>().OpenSession();    
            })
            .InSingletonScope();                          
}

I think this approach might suit a lot of projects better if your commands are dependency heavy.
What I like most is that handling architectural shoots will be easier. For example: right now, all behaviour is in my entities and in my commands; the segregation of application services and domain services is non-existent. And this works fine so far; I yet have to find a use case where I would benefit from more separation. If that would change in the future though, I can introduce abstractions, and concepts, without breaking consumer code, and without having to do awkward stuff managing the newly introduced dependencies.

As always, your thoughts are appreciated.

Self-contained commands with dependencies

2013-01-13T18:17:00.000+01:00

Also read: separating command data from logic and sending it on a bus

In October I looked at an architecture that limits abstractions to solely commands and queries. In that post, I had some infrastructure that looked like this.

public abstract class Command
{
    public abstract void Execute();
}
public abstract class Query<T>
{
    public abstract T Execute();
}
public interface ICommandHandler
{
    void Execute(Command command);
}
public class CommandHandler : ICommandHandler
{
    public void Execute(Command command)
    {
        command.Execute();
    }
}
public interface IQueryHandler 
{
    T Execute<T>(Query<T> query);
}
public class QueryHandler : IQueryHandler
{
    public T Execute<T>(Query<T> query)
    {
        return query.Execute();
    }
}

Commands and queries are both accompanied by their specific handler. In this example, the handler does nothing but invoking the command or query. In reality, you want your handlers to do a little more. For example: provide the commands and queries with a context to work with, add logging, handle your unit of work and all of that good stuff.

Let's look at the infrastructure, and particularly the handlers of a project that uses RavenDB to store its data.

public abstract class Command
{
    public IDocumentSession Session { get; set; }
    public abstract void Execute();           
}
public abstract class Query<T>
{
    public IDocumentSession Session { get; set; }
    public abstract T Execute();
}
public class CommandHandler : ICommandHandler
{
    public void Execute(Command command)
    {            
        var store = DocumentStore.Get();
        using (var session = store.OpenSession())
        {
            command.Session = session;
            command.Execute();
            session.SaveChanges();
        }
    }
}
    
public class QueryHandler : IQueryHandler
{    
    public T Execute<T>(Query<T> query)
    {
        var store = DocumentStore.Get();
        using (var session = store.OpenSession())
        {
            query.Session = session;
            return query.Execute();
        }
    }
}

The handlers take care of creating and managing the session, but also provide the commands and queries with a reference to the session.

An actual command could look like this.

public class ConfirmOrderCommand : Command
{
    private Guid _token;
    public ConfirmOrderCommand(Guid token)
    {
        _token = token;
    }
    public override void Execute()
    {
        var order = Session.Query<Documents.Order>().Where(x => x.Token == _token).First();
        order.ChangeStatus(Documents.Status.Confirmed);
    }
}

I really like this style of command and queries; very little ceremony. The downside though, is that you can't use constructor dependency injection. Like discussed in this post, you could split your classes in two parts: the handler and the data, and you would solve that problem.

I wasn't that keen on that approach. Also, now that I'm so accustomed to having fast in-memory integration tests with RavenDB, it's exceptional that I have the need to inject dependencies. I worked out an alternative which allows me to inject dependencies without having to put my data somewhere else.

Instead of injecting the dependencies through the constructor, we're going to use an Inject method. This is a convention; there is no interface that enforces this. Returning to our ConfirmOrderCommand, we'll add support for creating a new folder on the file system.

public class ConfirmOrderCommand : Command
{
    private IFileSystem _fileSystem;
    private Guid _token;
    public ConfirmOrderCommand(Guid token)
    {
        _token = token;
    }
    public override void Execute()
    {    
        var order = Session.Query<Documents.Order>().Where(x => x.Token == _token).First();
        
        _fileSystem.CreateDirectory(Path.Combine("D:\", order.Customer.Id));
        order.ChangeStatus(Documents.Status.Confirmed);
    }
    
    public void Inject(IFileSystem fileSystem) 
    {
        _fileSystem = fileSystem;
    }
}

We can now amplify the commandhandler to automatically resolve and inject the dependencies into our commands. The handler uses reflection to look for a method named Inject on the type. If this method exists, it will inspect the method for its expected arguments and try to resolve those, to finally invoke the Inject method with its resolved arguments.

public class CommandHandler : ICommandHandler
{
    private IKernel _kernel;
    public CommandHandler() { }
    public CommandHandler(IKernel kernel)
    {
        _kernel = kernel;
    }
    public void Execute(Command command)
    {
        if (_kernel != null)
            ResolveDependenciesIfNeeded(command);
            
        var store = DocumentStore.Get();
        using (var session = store.OpenSession())
        {
            command.Session = session;
            command.Execute();
            session.SaveChanges();
        }
    }
    private void ResolveDependenciesIfNeeded(Command command)
    {
        var method = command.GetType().GetMethod("Inject");
        if (method != null)
        {
            var parameters = method.GetParameters();
            var parameterInstances = new List<object>();
            foreach (var parameter in parameters)
            {
                var type = parameter.ParameterType;
                var instance = _kernel.Get(type);
                parameterInstances.Add(instance);
            }
            method.Invoke(command, parameterInstances.ToArray());
        }
    }
}

Consumers now don't have to care about the dependencies; they just have to be registered in the container. In the tests however, we can now explicitly inject mocks or stubs, and take advantage of having discoverable dependencies though the Inject convention.

[TestClass]
public class When_confirming_an_order
{
    private Mock<IFileSystem> _fileSystem;
    [TestInitialize]
    public void When()
    {
        DocumentStore.InitializeEmbedded();
        var cmd = new ConfirmOrderCommand("_token_");
        
        // Inject mock
        _fileSystem = new Mock<IFileSystem>();    
        cmd.Inject(_fileSystem.Object);
        new CommandHandler().Execute(cmd);
    }
    [TestMethod]
    public void the_status_is_changed_to_confirmed()
    {
        ...
    }
    
    [TestMethod]
    public void a_new_folder_is_created()
    {
        _fileSystem.Verify(...);
    }
}

Although I haven't really gone the distance with this implementation - I only have one command that has extra dependencies, I find this technique showing lots of promise. You get the leanness of self-contained commands and queries, while you still allow discoverable dependency injection by convention, supported by a tiny bit of infrastructure in the handlers.

I'd like to hear your opinion.

Keeping your AppHarbor application pool alive

2013-01-06T21:18:00.000+01:00

By default, IIS will shut down your application pool when it has been idle for more than 20 minutes. This is annoying when your website is only visited sporadically; visitors might not have the patience to wait for your application pool to spin up again. When you're running your own machine, you can higher or disable the idle-timeout, but when you're running on a cloud service like AppHarbor you can't.

One solution is to frequently make a request yourself to keep the application pool alive. You can use a third party service (like Pingdom or StillAlive), but chances are you don't want to take an extra dependency for something that trivial.

AppHarbor contains the required infrastructure to do this yourself: background workers and scheduling.

First create a new Quartz job which makes a request to your web application when it's invoked.

public class KeepAliveJob : IJob
{
    public void Execute(IJobExecutionContext context)
    {
        using (WebClient client = new WebClient())
        {
            client.DownloadString("http://your_webapp.com");
        }
    }
}

Then schedule your job to be triggered every 19 minutes or so.

var keepAliveJob = JobBuilder.Create<KeepAliveJob>().Build();
var keepAliveTrigger = TriggerBuilder.Create()
                .WithSimpleSchedule(x => x.WithIntervalInMinutes(19).RepeatForever())
                .Build();
scheduler.ScheduleJob(keepAliveJob, keepAliveTrigger);    
scheduler.Start();

And that should be it; you're now running your own ping service.

2012 Annual Review

2012-12-31T15:58:00.001+01:00

It's that time of the year again where I take some time to look back on the year passed and peek at the year ahead. This is mostly a post I like to write up for myself, forcing reflection.

The biggest personal changes this year were moving out of my parents' place, and my girlfriend's decision to study medicine after graduating as a Master of Arts earlier this year. I had no idea how things would turn out once we lived together, but so far we have been doing great.

My career

September marked one year working for Euricom. I'm still at my first client, and while I definitely regularly have my share of the enterprise blues, I also learned a lot working on several projects: from maintaining legacy to leading green fields. Fairly, I often doubt if traditional enterprise life is it for me though.

Blog

I published 66 posts this year, which seems to be consistent with previous years.

While writing has turned into a habit over the years, I often struggle with deciding what I should write about. Any reason is probably a good one though; documenting, sharing opinions and experiences, or just writing for the sake of writing... But I still feel like I should add value and contribute something worthwhile when I spend time on this blog.

The topics haven't changed that much this year. I still write often on Web technologies (ASP.NET, NancyFx and REST) - something I didn't get to spend much time on at work this year, but I also wrote about architecture, testing and NoSQL. Next to those topics, I also penned some of my experiences gained doing experimental side projects.
In 2013, I intent to get more out of my comfort zone, and to also write down thoughts which are not that strongly held.

Page views have more than doubled (173k) compared to 2011, which I'm humbled by, but I seem to care less about that than a little while ago. It had even been over a month since I logged into Google Analytics.

Community

It very much depends on what I'm doing that day, but I still spend quite some time on Twitter. I often blurt out thoughts on software, and regularly, someone who knows better than me replies, and challenges my looks on software. That's how I get value out of Twitter. That, and animated GIFs of course.

In real life, I went to see a few interesting speakers at local user groups, and gave a talk myself at Web.NET Europe Milan - international speaker, wooh. I want to applaud the organization of Web.NET Europe once more; they did a great job.

In general, I started to value artificial sponsored user groups less. They have their place, and they do add value, but it's not something that I feel I need to be a part of. Having some beers with a few likeminded developers suits me better.

Travelling

Buying an apartment and all of that, we didn't get to make any epic travels this year. I got to visit three different countries though: Italy, Tunisia, Italy again, and France. Two out of four were sponsored by my job; not bad.

We're setting out for a big trip again in 2013. We fell in love with America two years ago, so we've been researching the South-West again - but closer to the Mexican border this time. Other destinations are still open for discussion though.

Projects

I spent a considerate part of 2012 building small things on the side. This didn't result in anything useful, but most importantly, it got me in the habit of working on my own things, and I put together a toolset which enables me to ship things fast (NancyFx, Twitter Bootstrap, MongoDB, AppHarbor, ...).

The first project I did in 2012 was Docary. This was supposed to be something between a timesheet and a diary. This idea quickly died after I couldn't commit to using it myself. Technically I played with early versions of jQuery mobile and EF, and learned why you shouldn't overlayer your MVC application - or any web application for the matter.

For the second project, I and Davy Brion worked on TopDevLinks. This was supposed to be a linkblog implementation on top of ASP.NET MVC, CQS and MongoDB. We both lost interest in the concept eventually.

I made my girlfriend a petite portfolio site built on top of NancyFx, the FlickR API, and some jQuery plug-ins. Right now, it's actually running as a static site on GitHub pages though.

I already blogged about the next project: Kill Long Meetings. This one is still around, but quickly floundered into forgetfulness after its five minutes of Twitter fame.

The last two things I worked on this year were my Antwerp Open Data submissions. I jotted down some technical details here. The Nancy.AspNetSprites.Razor package also sprouted out of that effort (almost 100 downloads!).

Now that I look back at it, I did write quite some code after my hours this year... Next to being in the moment of building and shipping stuff, I genuinely believe it makes me better at my job.

I hope to be able to do as much in 2013, but to also let it be something worthwhile. I'll be actually already shipping something new in a few weeks if all goes well.

Sport/Health

The biggest gift I gave myself health wise this year, was giving up on smoking; I lit my last one over six months ago!

Moving to Antwerp, and not having any area to put my weights, I've joined a gym again, where I'm doing a two times a week compound routine. That should make me stronger again; I've lost a lot of beef since picking up running three years ago.

I haven't given up on running - living close to the city park helps. I just missed the 1000km (or 621 miles) milestone this year, which is kind of lame, but I'm far from complaining.

I don't know what I hope for in this area in 2013. It's starting to dawn on me that, even though those goals I set for myself have pushed me further than I thought I was capable of - I couldn't jog 1km without a break three years ago, they are not that important. It eventually comes down to nourishing your body, and making use of it in every way possible while you still can.

Conclusion

Earlier this week, I strolled down the main street of Brussels and crossed a beggar with one leg and no arms. Next to feeling extremely sorry for that man, I felt truly blessed; I'm healthy, loved, have more than I need, and I get to do what I like for a living. I need to remind myself more of that in 2013 - I can be a whiny bitch.

I plan to keep course in 2013, but as a better man.

How was your year? What do you plan for in 2013?

Now, if you'll excuse me, I'm going to help a hand in preparing the festivities before the girlfriend really gets in a faul mood.

Holy crap, when I sat down to write this, I didn't expect to write a whole book. I'm wondering how many people actually made it down here.

2012's most read posts

2012-12-23T15:53:00.000+01:00

I look forward to writing this post each year; it's by far the easiest one to write, and it provides me with an occasion to look back on previous years (2009, 2010, 2011). It's entertaining - and shameful too - to see what kept me busy back then, which opinions I held, and which technologies and techniques are still relevant in the meanwhile. Anyways, here it goes.

The most read post this year, with 37.797 page views, is How a web application can download and store over 2GB without you even knowing it. I just ran this experiment again with the latest Chrome build, and apparently the Chrome team hasn't addressed these concerns in the meanwhile; the browser still behaves exactly the same. I'm guessing usage stats might prove that it's not worth the effort.

The second most read post, with 14.371 page views, is Learning: the Hacker Way. I still firmly stand by this one today. Passive learning can only take you so far; it's crucial to supplement passive learning with a high volume of getting your hands dirty.

With 14.085 page views, Commuting: have you done the math? comes in third. Commuting wastes a good part of your life; optimizing your commute can significantly improve the quality of your average day. Readers shared some interesting experiences in the comments on this one.

Having 5.215 page views, the post Why I will always love RSS is the penultimate item in this list. This writing is basically a rant on how, although I somewhat understand the motivation behind public companies fencing their gardens, it is sad to see that even peers seem to advertise the death of RSS, while it's RSS that has enabled me to follow my favorite sources of information without having to rely on others to tell me what to read, and without having to be connected the whole damn time.

With 1000 views less, making it the last item in the list, is the post HTML5 Offline Web applications as an afterthought in ASP.NET MVC. It was this post that landed me a paid article on InfoQ. I should revisit this topic again, I'm curious to see how many websites are already making use of this HTML5 super cache to improve performance, or to truly provide offline access.

Thank you for reading in 2012!

Released: My Antwerp Open Data submissions

2012-12-16T16:44:00.000+01:00

A little while ago the city of Antwerp released their Open Data initiative, and it included a meetup where you could show something you built, build something on the spot, or pitch your ideas. When I first heard of the initiative I had nothing going on the side, and was looking for something tangible I could build to try out a few technologies. I couldn't come up with an original idea, and ended up building two web applications using the Open Data datasets: Culture Mosaic, and Where to pee in Antwerp?
Since I was bedridden due to the flu, and my ideas were not that great, I eventually ended up not going to the meetup, but I figured I would share some stuff I picked up along the way here.

Culture Mosaic

The first web application uses the culture dataset to display a masonry of cultural attraction images.

The images available vary in size, and some turned out to be quite large. If I wanted to display all the images at once, I had to make them smaller; resize them. For this task, I found the ImageResizer library on Nuget. Although you can reference remote images using an ImageResizer extension, I chose to download the images locally, to then resize them and to finally cache them on disk.

I run these tasks in parallel and wait for all of them to finish, using the TPL.

private void TryToDownloadAllImages(Culture culture)
{
    var tasks = new List<Task>();
    foreach (var item in culture.Items)
    {
        var localPath = Path.Combine(_pathConfiguration.ImagesFolderPath, item.ImageName);
        if (!File.Exists(localPath))
            tasks.Add(CreateImageDownloadTask(item.Image, localPath));
    }
    Task.WaitAll(tasks.ToArray());     
}

Downloading a remote file is straightforward using the WebClient class. Right after downloading the image, I use the ImageBuilder singleton to resize the image to a width of 340px.

private Task CreateImageDownloadTask(string imageUrl, string localPath)
{
    return Task.Factory.StartNew(() =>
    {
        using (var client = new WebClient())
        {
            try
            {                        
                client.DownloadFile(imageUrl, localPath);
                ImageBuilder.Current.Build(
                    localPath, 
                    localPath, 
                    new ResizeSettings("maxwidth=340"));
            }            
            catch (ArgumentException) 
            {
                // filename malformatted; swallow
            }
        }
    });                    
}

Now that I have my resized images, I can render them in my view. This is NancyFx with the Razor view engine by the way.

<div id="loading">
    Loading..
</div>      
<div id="container" style="display:none;">
    @foreach (var item in Model.Items)
    {               
       <a href="@item.Url">
          <img src="/Content/images/@item.ImageName" alt="@item.Name" title="@item.Name" />
       </a>             
    }
</div>

To achieve the masonry effect, I used this jQuery plug-in. The one tricky part here is to only initialize the effect after all the images have really loaded. For this, I used the imagesLoaded plug-in.

$container.imagesLoaded(function () {
    $loading.hide();
    $container.show();
    $container.masonry({
        itemSelector: 'img'
    });
});

One last thing I wanted, was to center the masonry on screen, but since I didn't want to use a fixed width, I had to dynamically recalculate the optimal width each time the window is resized.

var setContainerWidth = function (container) {
    var imageWidth = 340;
    var bodyWidth = $("body").width();
    var numberOfImagesNextToEachother = Math.floor(bodyWidth / imageWidth);
    var optimalWidth = (imageWidth * numberOfImagesNextToEachother) + (numberOfImagesNextToEachother * 4);
    container.css("width", optimalWidth + "px");
};
$(window).resize(function () {
    setContainerWidth($container);
});

Since we have the width of the images (340px), we can use trivial math to calculate the optimal amount of width we can consume.

You can find the working bits on AppHarbor. Source is on GitHub.

Where to pee in Antwerp?

The next application is a mobile web application that uses the public toilets dataset to show the nearest toilets based on your location.

I wanted to introduce a constraint client-side of having to work with a somewhat crooked dataset, that's why I just return the dataset using Nancy as is.

public ToiletsModule()
{
    Get["/Toilets"] = p =>
    {
        var response = Response.AsFile(@"Content\json\toilets.json");
        response.ContentType = "application/json";
        return response;
    };
}

At the client, I'm using jQuery Mobile and angular.js. It had been a while since I had done something with jQuery Mobile, but this version felt very solid (compared to the beta versions I played with before). Turns out though, that to make jQuery Mobile and angular.js play nice, you have to include the jQuery Mobile Angular adapter. The size of all these libraries unminified was a performance hog; Cassette to the rescue.

The first thing you do with angular, is defining a module. A module instantiates, wires and bootstraps the application.

var toiletApp = { };        
toiletApp.module = angular.module('toiletModule', []);

To separate concerns, and make things testable, angular lets you define services for your application using a factory. By using a factory you also make your service an injectable dependency for other parts of your application.
I added a toilet service to this module which makes the ajax call, sorts the results based on nearest by, to eventually make them available to the caller by invoking a callback. The $http dependency is one of angular's core services that lets you do ajax requests.

toiletApp.module.factory('toiletService', function ($http, Toilet) {
return {
    getToilets: function (onSucess, onError, coordinates) {
        $http
            .get('toilets')
            .success(function (data) {
                var toilets = [];
                for (var i = 0; i < data.openbaartoilet.length; i++) {
                    var toilet = data.openbaartoilet[i];
                    var ourToilet = new Toilet(
                        toilet.omschrijving,
                        toilet.point_lat,
                        toilet.point_lng,
                        toilet.betalend === "niet betalend" ? false : true,
                        toilet.straat,
                        toilet.huisnummer);
                    toilets.push(ourToilet);
                }
                if (coordinates) {
                    toilets.sort(function (a, b) {
                        var aDistance = a.calculateDistance(coordinates);
                        var bDistance = b.calculateDistance(coordinates);
                        if (aDistance < bDistance)
                            return -1;
                        if (aDistance > bDistance)
                            return 1;
                        return 0;
                    });
                }
                onSucess(toilets);
            })
            .error(function() { onError() });
    }
};

You might have noticed the Toilet dependency; into this class I project the items returned from the ajax call.

toiletApp.module.factory('Toilet', function () {               
var Toilet = function (name, lat, lng, isPaying, street, houseNumber) {
    var lastCalculatedCoord;
    var lastCalculatedDistance;
    this.name = name;
    this.latitude = lat;
    this.longitude = lng;
    this.isPaying = isPaying;
    this.street = street;
    this.houseNumber = houseNumber;
    this.lastCalculatedDistance = function () {
        return lastCalculatedDistance;
    };
    this.calculateDistance = function (coordinates) {
        if (coordinates) {
            if (lastCalculatedCoord == coordinates) {
                return lastCalculatedDistance;
            }
            var currentPosLatLon = new LatLon(coordinates.latitude, coordinates.longitude);
            var destinationPosLatLon = new LatLon(this.latitude, this.longitude);
            var dist = Math.ceil(currentPosLatLon.distanceTo(destinationPosLatLon) * 1000);
            lastCalculatedCoord = coordinates;
            lastCalculatedDistance = dist;
            return lastCalculatedDistance;
        }
        return null;
    };
    this.getMapUrl = function () {
        var coord = this.latitude + "," + this.longitude;
        return "...";
    };
};

The Toilet class also has a method to calculate the distance from the item to any given coordinate. I first wanted to make use of the Google Distance Matrix API to calculate the actual travel distance, but that didn't really work out to well for more than 100 items; you can't batch your requests, and the free version of the API has a threshold on the amount of queries. I turned to calculating the great-circle distance instead, using this library.

Since we want to make use of the client's coordinates, I also defined a geo service, which makes use of the HTML5 GeoLocation API to query for the user's coordinates.

toiletApp.module.factory('geoService', function() {
    return {
        getLocation: function (onSucess, onError, onNotAllowed) {
            if (navigator.geolocation) {
                navigator.geolocation.getCurrentPosition(
                    function (position) {
                        onSucess(position);
                    },
                    function (err) {
                        onError(err);
                    }
                );
            } else {
                onNotAllowed(message);
            }
        }
    };
});

After defining these pieces, I injected them into the ContentController. The controller is responsible for augmenting angular's scope, which is the connection to our view, our model. Properties and methods are made available to the view by augmenting the $scope variable. There is quite some not that interesting tinkering going on in the controller, so I left it out for brevity.

toiletApp.ContentController = function ($scope, geoService, toiletService) {
    $scope.toilets = [];
    ...
}

After setting ng-app and ng-controller, we can make use of angular's directives and filters to render our page. Directives are - as the documentation puts it - a way to teach HTML new tricks. Think of it as HTML helpers. Filters are used to format data.

The view binds each toilet to a new collapsible div. The content of the div shows some basic information, and a button which shows the toilet on a map.

<html ng-app="toiletModule">
    <head>
        ....
    </head>
    <body>
        <div data-role="page" id="home" ng-controller="toiletApp.ContentController"> 
            
            <div data-role="header">
                <h1>Where to pee in Antwerp?</h1>
                <a href="#" data-icon="refresh" ng-click="reload()">Reload</a>                
            </div> 
            
            <div data-role="content">
                
                <div class="content-primary">    
                    <p>{{ message }}</p>
                   
                    <div data-role="collapsible" 
                         ng-repeat="toilet in toilets | limitTo: itemsLimit()">
                        <h3>
                            {{ toilet.name }} 
                            ({{ toilet.lastCalculatedDistance() | distance }}) 
                            {{ toilet.isPaying | isPayingDollarSign }} 
                        </h3>                        
                        <p><strong>Name: </strong>{{ toilet.name }}</p>
                        <p><strong>Distance: </strong>{{ toilet.last..() | distance }}</p>
                        <p><strong>Paying: </strong>{{ toilet.isPaying | isPayingYesNo }}</p>
                        <p><strong>Street: </strong>{{ toilet.street }} </p>
                        <p><strong>Housenumber: </strong>{{ toilet.houseNumber }} </p>
                        <a href="{{ toilet.getMapUrl() }}" data-role="button">
                            Show location on a map
                        </a>
                    </div>                     
                             
                    <div ng-show="hasMoreItemsToShow()">
                        <button ng-click="showMoreItems()">Show more toilets</button>    
                    </div>        
                </div>        
            </div> 
            
            <div data-role="footer">
                <h4>Made possible by Antwerp Open Data</h4>
            </div> 
        </div>
    </body>    
</html>

You might notice how I used filters to format the isPaying and distance property. These are also defined in the module.

toiletApp.module.filter('isPayingDollarSign', function () {
    return function (value) {
        return value ? "$" : "";
    }
});
toiletApp.module.filter('isPayingYesNo', function () {
    return function (value) {
        return value ? "Yes" : "No";
    }
});
toiletApp.module.filter('distance', function () {
    return function (value) {
        return value ? value + "m" : "";
    }
});

You can find the working bits on AppHarbor. Source is on GitHub.

Summary

I can only applaud the Open Data initiative; my hope is that the datasets get more interesting though. Which also makes me wonder if the government is even measuring the more interesting stuff?

All in all, these two applications aren't very innovative. Yet, they did give me the opportunity to experiment with some stuff I can't play with on the day job (yet). I'm always amazed how quickly you can slap together a small application using the available building blocks out there. I'm most content with picking up angular.js; I will be advertising this framework in a professional context from now on.

Show More pagination with angular.js

2012-12-12T19:57:00.000+01:00

I built my first application with angular.js over these last few weeks (not during business hours), and although I still have lots and lots to discover and learn, I think I somewhat grasp the basics.

In the application I built, I had to implement paging because rendering all the items at once was too slow on mobile devices (on my Windows Phone 7 anyways). The paging variant I decided on was the 'Show More' technique.

Let me walk you - as an introduction to Angular - through a simple application that uses paging for a list of 24 items.

We start by defining a module. In an Angular module, you should instantiate, wire and bootstrap parts of your application.

var module = angular.module('module', []);

We can already bind the module to our empty view.

<div ng-app="module">
</div>

In our module we want to define an itemService which will return an array of 24 items. By using a factory method we make our service an injectable dependency for other parts of our application.

module.factory('itemService', function() {
    return {
        getAll : function() {
            var items = [];
            for (var i = 1; i < 25; i++) {
                items.push('Item ' + i);                       
            }
            return items;
        }
    };              
});

Now that we have a service that can give us items, we want to work towards actually rendering these items. To do that, we'll have to define a controller. An Angular controller lets you augment an instance of Angular's scope, which in its turn serves as your connection to the view.

ListController = function($scope, itemService) {
    $scope.items = itemService.getAll();    
};

Notice how we use dependency injection to inject an instance of the itemService.

All our items are available on the scope now, but we still have an empty view. Let's fix that.

<div ng-app="module" ng-controller="ListController">
    <ul>
       <li ng-repeat="item in items">
          {{ item }}        
       </li>               
    </ul>
    <button>Show more</button>    
</div>

Items are being rendered making use of Angular's built-in ng-repeat directive. Directives are - as the documentation puts it nicely - a way to teach HTML new tricks. MVC folks may think of it as HTML helpers.

What's left to do in this example, is what we initially set out to do; implement paging. For that, we'll first add some extra variables to the controller, and some new methods on to the scope. These should be self-descriptive.

ListController = function($scope, itemService) {
    var pagesShown = 1;
    var pageSize = 5;
    $scope.items = itemService.getAll();
    $scope.itemsLimit = function() {
        return pageSize * pagesShown;
    };
    $scope.hasMoreItemsToShow = function() {
        return pagesShown < ($scope.items.length / pageSize);
    };
    $scope.showMoreItems = function() {
        pagesShown = pagesShown + 1;         
    };
};

We can now make use of those new methods on the scope in our view.

<div ng-app="module" ng-controller="ListController">
    <ul>
       <li ng-repeat="item in items | limitTo: itemsLimit()">
          {{ item }}        
       </li>               
    </ul>
    <button ng-show="hasMoreItemsToShow()" ng-click="showMoreItems()">Show more</button>    
</div>

We made use of directives to add behaviour to the 'Show more' button and used the limitTo filter to limit the number of items rendered. Filters are used to format display data.

Here is the full jsFiddle.

I'm pretty sure you can abstract paging into a reusable component, but I thought this scenario shows a good bunch of basic Angular concepts.

So what are your thoughts on Angular? Pretty slick, right? Or are you already using Angular?

It's not cake we are baking

2012-12-09T16:08:00.000+01:00

I recently watched a talk on Vimeo where Christin Gorman talks about how cookie dough relates to Hibernate; why use the generic, bloated and one-fits-all solution when you can mix together your own yummy cookie dough? We should aspire to be the Gordon Ramsey of software, not the college student who can only cook Ramen noodles. If you haven't watched or listened to her talk, you should; it's only a few minutes long, and she brings it really well. Go ahead, I'll wait.

When I first watched her talk, it rubbed me the wrong way, but I couldn't yet figure out the flaw in her plea. It recently dawned upon me that we are not in the business of baking cakes, but we are in the business of serving hungry people. And those people really don't care which ingredients were used to bake your cake; they only care about how it tastes, how fast it's served, and how much they're paying for it. And although the romantic in me really sympathizes with her view, the realist just can't. Over these few years only, each and every attempt of a hand-rolled ORM I've seen, ended with a horrible aftertaste. There is no way that you spend as much time building a good API from scratch as you do setting up an existing one. Writing and maintaining your own ORM is not a trivial thing to do, and there are already so many flavors* - from grandma's recipe to decadent supreme deluxe - to pick from, that it's hardly ever a justified decision. Read the package and choose the option that satisfies your daily caloric needs, and go from there. You can still sweeten, or swap some ingredients, before serving. And if you really can't resist - which is perfectly healthy, bake all the cake you want, at home. Maybe, one day, you can bring a successful one to work.

* NHibernate, Entity Framework, Dapper, Massive, Simple.Data, LLBLGen Pro, OpenAccess, SubSonic ...