log4net PatternString

I’ve been using log4net since it was first available for Mono and barely know anything about it. To a certain extent you’ve got to love it because of that – it just works! The project is great, but development appears somewhat stagnant. Either way, I’ve never needed to know much more than basic configuration and how to get it monitored even though I’ve used it for daemons, client applications, Add-ins, etc.

Today however, I learned something! A while ago I needed to have the RollingFileAppender log to a varying location depending on Windows version (XP vs Vista/7). At the time I found posts showing me how to use variables within the “file” option like so:

<file value="${APPDATA}\Company\Product\logs\data.log" />

Not having looked at the log4net code this is either explicitly converted to a user’s roaming folder, or it is replaced as an environment variable. This works great for applications that should log per user, but what if I needed the “All Users” roaming folder on Windows? Neither ${COMMONAPPDATA} or ${ALLUSERAPPDATA} worked for me so I needed another option that would cut it.

Enter log4net.Util.PatternConverter

By extending this very simple class and implementing one method we can now do variable substitution. The syntax leaves a lot to be desired, but hey – it works!

<file type="log4net.Util.PatternString">
  <converter>
    <name value="sub" />
    <type value="SpecialFolderPathConverter, Assembly" />
  </converter>
  <conversionPattern value="%sub{ApplicationData}\Company\Product\logs\addin.log" />
</file>

With this setup now all I have to do is create the Assembly.SpecialFolderPathConverter class:

using System;
using System.IO;

namespace Assembly
{
    public class SpecialFolderPathConverter : log4net.Util.PatternConverter
    {
        protected override void Convert(TextWriter writer, object state)
        {
            // Option gets set to contents of {}
            if (String.IsNullOrEmpty(this.Option))
                return;

            try {
                Environment.SpecialFolder sf = (Environment.SpecialFolder)
                    Enum.Parse(typeof(Environment.SpecialFolder), this.Option, true);

                writer.Write(Environment.GetFolderPath(sf));
            } catch (Exception) {
            }
        }
    }
}

As you can probably see this class takes any string within the {} brackets following %sub and replaces it using Environment.GetFolderPath which is quite simple, and helpful.

Thank you log4net!

Mono Embedded – Your Own .NET Plugin

I’ve embedded either for fun or work a number of languages to see what hosting them in C is like – both for performance and ease of marshalling. So far I’m impressed with the internals of Mono, but feel like there’s a lack of proper GOTCHA’s documented anywhere. Some are spread out around the web, but most don’t directly state key points. As such, I’m going to be posted a few as I come across them.

WARNING: Some of this may not make complete sense without reading http://www.mono-project.com/Embedding_Mono first.

Creating Objects
Like the above docs day, all you really need to do is:

/* get handle for class from assembly */
MonoImage *image = mono_assembly_get_image(MonoAssembly*);
MonoClass *klass = mono_class_from_name(image, "Namespace", "Class");

/* instantiate class and call .ctor (as they're nothing more than methods) */
MonoObject *obj = mono_object_new(MonoDomain*, klass);
mono_runtime_init(obj); /* call .ctor w/out args */

Reusing Objects
If you need to reuse this new MonoObject instance EVER I would highly suggest calling mono_gchandle_new(MonoObject*, pinned) otherwise the GC will get to it before you want it again – I GUARANTEE THIS. The reason this happens is because objects created from your C code have no real “SCOPE”, so they’re collected quickly just like they were local variables in a method. Read the following link, it’s helpful:

http://www.go-mono.org/docs/index.aspx?link=xhtml%3Adeploy%2Fmono-api-gchandle.html

As a side note, you can actually call methods on objects that have been collected as long as the methods usage of `this` is only limited to `this.GetType()` (or nearly static). Try it, you’ll get some really funny results!

Finding Methods
The MonoClass* struct contains (or points to) lots of lists of data. The methods of a class are essentially one big long list and each method (every single variation) is a different instance of MonoMethod*. Take a look at the following C# class:

namespace Test
{
	public class Foo
	{
		public Foo()
		{
		}

		public void Log(int num)
		{
		}

		public void Log(string msg)
		{
		}
	}
}

NOTICE: The Log method is overloaded!

One way to call Foo.Log(?) is like so:

/* get handle for class from assembly */
MonoImage *image = mono_assembly_get_image(MonoAssembly*);
MonoClass *klass = mono_class_from_name(image, "Namespace", "Class");

/* instantiate class and call .ctor (as they're nothing more than methods) */
MonoObject *obj = mono_object_new(MonoDomain*, klass);
mono_runtime_init(obj); /* call .ctor w/out args */

/* Get Log method from MonoClass that takes 1 argument
MonoMethod *m = mono_class_get_method_from_name(klass, "Log", 1);

/* create single argument */
void *args[1] = { mono_string_new(MonoDomain*, "Hello World"); };

/* last argument is to capture exceptions */
mono_runtime_invoke(m, obj, args, NULL);

Any guess what might have just happened? Well, we just passed a string to the Log(int) method because mono_class_get_method_from_name returns the first one it finds, and not the one based on the type. To fix this you could write your own method which loops through MonoMethod*’s and find the correct signature or use the helper methods:

#include <mono/metadata/debug-helpers.h>

MonoMethodDesc *d = mono_method_desc_new(":Log(int)", FALSE);
MonoMethod *m = mono_method_desc_search_in_class(d, klass);
mono_method_desc_free(d);

The mono_method_desc_* functions are extremely helpful and remove some of the crazy and changing internals – so I’d suggest using them for much of your calls.

Here are some further examples of the string descriptor for mono_method_desc_new:

  • Namespace.Test:Log(string) -> find Log method in Test class (requires second argument to be TRUE)
  • :Log(string) -> find Log method which takes a string
  • :Log(XmlReader) -> find Log method which takes an XmlReader
  • :Log(My.Namespace.XmlReader) -> find the method which takes YOUR funky XmlReader

As you can see it covers the gamut – so thank you to whomever created that extremely useful class/struct/function-set.

~ Till next time!

Mono + SCGI

Over a year ago I started a SCGI daemon for Mono (scgi-mono-server). For those who don’t know, the “SCGI protocol is a replacement for the Common Gateway Interface (CGI) protocol. It is a standard for applications to interface with HTTP servers. It is similar to FastCGI but is designed to be easier to implement”.I stopped all development shortly after I started because lighttpd (the web server I cared about) required a very small patch to be useful. Having supplied the patch I waited around for it to be rolled into a release (1.4.20) and then waited for it to be picked up by Ubuntu, but it still hasn’t. So what brought it back to life? A silly bug: https://bugs.launchpad.net/ubuntu/+source/mod-mono/+bug/227781.

Some of our servers need both PHP5 & Mono to run side-by-side so one of our guys listed the options:

  • Pull mono & mod-mono 1.9.1 back-ports from a PPA (http://ppa.launchpad.net/directhex/ppa/ubuntu)
  • Patch and rebuild mod-mono 1.2.5 ourselves
  • Switch to FastCGI for PHP5, so we can use apache2-mpm-worker for mod-mono-server
  • Switch from Apache to lighttpd (and rebuild with our patch)
  • Switch from Apache + mod-mono to Apache + FastCGI or SCGI (requires patching lighttpd)
  • Switch from Apache + mod-mono to Apache + mod_proxy for XSP

Given our growing dislike for Apache & appreciation for lighttpd we decided on a phased approach:

  • Switch PHP5 to Apache + FastCGI
  • Switch Mono to Apache + SCGI
  • Run for a while (make sure everything works well)
  • Switch from Apache to lighttpd

We choose SCGI for Mono over FastCGI because:

  • FastCGI requires a lot of chatter between the client & server
  • Great incentive to finally give the server to Mono

With our approach in mind I started working on scgi-mono-server again, but with a new target – Apache. This shouldn’t have been very hard given that mod_scgi is provided by Quixote, but there were a few stumbling blocks. Apache, for whatever reason, has decided to break the SCGI specification (in my eyes). As stated in the Protocol: “[t]he format of the response is not specified” – meaning whatever the SCGI server sends back to the client (Apache, lighttpd, etc) should go back to the originator (web browser). Apache however, does not honor this.

The first line of any HTTP response (AFAIK) should be it’s Status Line:

HTTP/1.1 200 OK.

Apache’s mod_scgi requires that all lines until the body have a colon so that is can parse the header and update it’s internal data model. Therefore, our first line now has to be replaced by:

Status: 200 OK

Thankfully the Example section of the protocol showed me how to fix the issue. I’m tempted to patch Apache/mod_scgi but I’m a bit worried about how many SCGI servers require this broken code. So for now I’ve added a configuration option that can be put inside ASP.NET’s web.config to control the hack:

<appSettings>
	<add key="MonoServerApacheStatus" value="true" />
</appSettings>

I’ll be submitting the server back to Mono with some documentation after it goes through some solid developer testing on our side – so I hope someone enjoys!

Programming Tutorial: Comments

Comments are only effective if they threaten abuse.

/**
 * Note: The following array MUST be sorted in order for BinarySearch to work. I have taken the liberty of
 * creating it sorted to avoid this operation. If you really feel the need to break the manual sorting,
 * uncomment the array.sort below this, and then go out and stand in front of traffic. thank you.
 */

string[] allowed_items  = {"activate", "restore", "save", "update-relay", "verify-mount", "verify-restore"}; 

//Array.Sort(allowed_items);

if (Array.BinarySearch(allowed_items, list) >= 0) {
	/* -.v.- */
}

Discovered by Noah Massey.

RESTful Web Services in .NET (Part 2)

Now that I’ve finally got a bit of time I figured I should explain how we updated our old RESTful framework to be nice and slick. Instead of one method per class which handles all sub method variations we now have something much prettier and easier to maintain. The UserService found in the previous post can now look like:

public class UserService : RestService
{
	[RestMethod("GET", "^$")]
	public IList RetrieveAll()
	{
	}

	[RestMethod("GET", @"^(?<id>[0-9]+)$")]
	public User Retrieve(int id)
	{
	}

	[RestMethod("HEAD", @"^(?<name>\w+)$")]
	public User Exists(string name)
	{
	}

	[RestMethod("PUT", "^$", typeof(User))]
	public User Create(User user)
	{
	}

	[RestMethod("POST", "^$", typeof(User))]
	public bool Update(User user)
	{
	}

	[RestMethod("DELETE", @"^(?<id>[0-9]*)$")]
	public bool Delete(int id)
	{
	}
}

As you can see there are a lot of differences:

  • Heavy influence from newer APIs
  • Methods can return any Type
  • Methods can take 0 or more arguments of any Type
  • RestMethodAttribute has been added to match URLs against methods

Most of the code above should be self-explanatory, but in case it’s not here’s some help.

The method decorator [RestMethod("GET", "^$")] tells the framework that any HTTP GET requests against the UserService class without any trailing arguments (http://localhost/user.r) should be handled by RetrieveAll().

The method decorator [RestMethod("GET", @"^(?<id>[0-9]+)$”)] tells the framework that any HTTP GET request against the UserService class which ends with a number only (http://localhost/user.r/1, http://localhost/user.r/5000, etc) should be handled by Retrieve(int id). The cool thing here is that the regex group will provide a match to the argument name so our function will be called with whatever number the URL ends with. So calling http://localhost/user.r/25 where the following is the executed method will result in?

[RestMethod("GET", @"^(?<id>[0-9]+)$")]
public bool Retrieve(int id)
{
	return id == 25;
}

TRUE!! Pretty simple, and very cool. Another nice feature is when the HTTP request contains a body like a POST or PUT would. So POSTing to http://localhost/user.r with a body of:

{'user': {
	'first_name': 'Matthew',
	'last_name': 'Metnetsky'
}}

would call the method decorated with [RestMethod("POST", "^$", typeof(User))] and it’s user argument would actually be set assuming the class implemented a JSON serializer. All of the URL/body mappers are definable by implementing Binder and listing them in the web.config against a Content-Type. By default there are Binders for form variables, JSON, XML, raw streams, and a few more.

So why might I be discussing the external API of a closed library? To PUSH innovation in the current public ones. By taking these ideas and creating an “open” version I’m sure I’d be breaking my contract so I’ve got two options:

  1. Expose our public API to entice and PUSH others to innovate
  2. Get a lot of feedback which might enable me to get our library opened

So…. let me know…. ;-)

RESTful Web Services in .NET (Part 1)

A couple of years ago I tossed together a RESTful IHttpHandlerFactory for .NET. It wasn’t amazing, but it was being used long before Microsoft et all decided to create there own. Instead of extending Page you extended RestService and implemented whatever methods you wanted.

public interface IRestService
{
	void Head(RestEventArgs args);
	object Get(RestEventArgs args);
	object Put(RestEventArgs args);
	object Post(RestEventArgs args);
}

As you can probably guess the request’s HTTP Method was transformed into the function name and executed accordingly. If the method wasn’t “known” we would then use reflection to find the method for execution. Not to fansy as you can see, but it worked.

If a request was made for http://localhost/app/users.r the IHttpHandler factory would load users.r and it would tell us what class you assigned to it (much like @Page CodeBehind/Inherits in .aspx). From there we would instantiate the class and call the appropriate method. Whatever you returned would be marshalled for wire transfer based on it’s type.

	class User : IXmlSerializable
	{
		/* ... */
	}
    
	public class UserService : RestService
	{
		public object Get(RestEventArgs args);
		{
			return new User(5, "Matthew", "Metnetsky");
		}
	}
}

Calling http://localhost/app/users.r would execute UserService::Get(..) which would return a User instance. The results are then inspected for a few different interfaces like IXmlSerializable, IJsonSerializable (one of ours), etc. Based on what is supported by the Type, we then check if the requester supports it by interogating their “Accepts” header. This way each requester can receive what they understand how to handle (for instance – don’t send JSON to AS3). The system really worked well, and it’s simplicity made it damn fast.

Like all frameworks however, once it was actually in use we were able to see the pitfalls and issues. Take a look at the following URLs:

  • http://localhost/app/users.r
  • http://localhost/app/users.r/1
  • http://localhost/app/users.r/matthew

In order to handle all of the urls above the Get(RestEventArgs) method turned into one big switch/if/else conditional. Some people made it prettier than others, but the result was the same. So about 14 months ago we upgraded our framework – check back soon to see how we did it, and why I’m bringing it up now.

PMS2 – Coming Soon to an Interweb Near You

A long, long time myself and two friends were working on a college project where the entire goal as I saw it was to “create” stuff. Mono was a wee lad back in 2003, but I still preferred it to Java any day of the week. So myself, Phil Tricca, and Joseph Scaduto created a whole framework for creating what are now called “learning communities” in Mono/Gtk#/mod_mono.

One of the pieces we needed was a databases abstraction layer (we had grand ideas) so we first sought to flush out the ObjectSpaces API for Mono. Sadly, Microsoft couldn’t get their act together so we built our own. So in honor of Microsoft we called it Phil-And-Matt Spaces, shortened to PMS. After it’s original debut it sat on the selves for quite some time until I started working for Konica Minolta Business Solutions and was tasked with creating Printgroove. PMS hit the spot (man that sounds funny) – it was far from bad, but it certainly wasn’t awesome. At the point of its resurrection it needed a face lift, but never got one.

It’s now been years since Printgroove got started and PMS has continued to work well for it and numerous other projects – but I’ve got an itch that must be scratched. So while working on a small web service only project I’ve been adding in a couple dozen extra hours to mature PMS – or basically throw away it’s entire user facing API.

Here’s a sneak peak of whats to come:

// - Pull IDbConnection from pool based on unique name (from config file)
// - This is useful if you have multiple databases to chat with
// - Not specifying a name will load the one marked default or the first one found
using (DbBroker cxt = new DbBroker("name-mapping-to-config-connection-string")) {

    // SELECT member.* FROM member WHERE username LIKE '%a%' ORDER BY id
	// Enumerate through results without storing Member objects in container
	foreach (Member m in cxt.Query<Member>().Like("username", "%a%").OrderBy("id").Exec())
		Console.WriteLine(" : " + m);
    
	// Use raw SQL and ignore chainable methods for generation
	// Store results in List<Member> and then iterate through it
	string sql = "SELECT member.* FROM member where username LIKE '%a%' ORDER BY id";
	foreach (Member m in cxt.Exec<Member>().Objects<List<Member>>(sql))
		Console.WriteLine(" : " + m);

	// get first member
	Console.WriteLine("first: " + cxt.Query<Member>().Exec().First());

	// a short-cut to get the first member since we aren't filtering etc
	Console.WriteLine("first: " + cxt.Exec<Member>().First());

	// SELECT COUNT(member.*) FROM member WHERE id > 50000
	// Use generics to cast the output of ExecuteScalar to an Int32
	Console.WriteLine("count: " + cxt.Query<Member>().Filter("id > 50000").Exec().Count<int>());

	// Directly call ExecuteScalar but use generics to cast the output
    sql = "SELECT id FROM member WHERE username='mimetnet'";
	Console.WriteLine("scalar: " + cxt.Exec<Member>().Scalar<int>(sql));

	// Directly call ExecuteReader() with our generated SQL
	// SELECT id,username FROM member WHERE id > 68665
	using (IDataReader reader = cxt.Query<Member>()
									.GreaterThan("id", 68665)
									.SetColumns("id,username")
									.Exec()
									.Reader()) {
		while (reader.Read()) {
			Console.WriteLine("Member(id={0}, name='{1}')", reader[0], reader[1]);
		}
	}
}

Please send me your thoughts/concerns/nightmares.