Around thursday morning or so (maybe it was wednesday) the site was spewing so many 500 errors that somebody who I'm not sure I know where I know them from, emailed me saying "dude, site's broke." After checking the error logs, I found a lot of errors that were all like this:
fcgi-server re-enabled: 0 /tmp/fastcgi-mono-server
backend is overloaded; we'll disable it for 2 seconds and
send the request to another backend instead: reconnects: 0 load: 130
fcgi-server re-enabled: 0 /tmp/fastcgi-mono-server
backend is overloaded; we'll disable it for 2 seconds and
send the request to another backend instead: reconnects: 0 load: 130
After diagnosing the problem and kicking the server again, I decided that a couple of tips on the wiki page for Mono's FastCGI & Lighttpd had done me in, the first being about the FastCGI handler's max-procs configuration variable:
"max-procs" specifies the maximum number of servers to spawn. Because ASP.NET stores session specific objects, I am unsure of how applications would react if switching from one server to another, or if Lighttpd bonds a single server to a client. As such, I highly recommend keeping this value as "1" to avoid any conflicts.
Fortunately Urlenco.de doesn't really need any session information, so I did what Emeril and Apache admins are both familiar with doing, I kicked it up a notch (to about 10). After kicking the server one more time, this time with "max-procs" > 10 I watched the load on my little 1U server spike up to 30. While every terminal I had became so sluggish I could barely interact with the machine, I managed to open up "top(1)" and see what processses were royally screwing my machine. Turns out it was 10 instances of Mono, all trying to digest an ASP.NET site at once, all competing for the meager resources available. It seems that the Mono FastCGI server will process and compile your entire ASP.NET web application as soon as the FastCGI server is bootstrapped and accepting requests. Fortunately pushing new code to the site gets updated on the next HTTP request, so the number of times you'll have to kick (i.e. restart) the Lighttpd server should be minimal and you won't have to incur the huge performance penalty that often (I've since changed max-procs to 4).
I also went against some of the other advice on the wiki page
To overcome these problems, the recommended method for processing files is to send all requests directly to the FastCGI Mono Server.
By effectively passing every single request off to the Mono FastCGI Server you can avoid exposing some internal ASP.NET resources that should be interpreted and not sent over the wire, this seems to be poor practice as far as Lighttpd and FastCGI are concerned. Lighttpd is a very good, high performance HTTP server and should be allowed to do it's job, whereas FastCGI servers merely serve as a means for executing server-side pages, returning markup, etc. To avoid passing every single request off to the FastCGI server, I merely setup the FastCGI handler for .aspx pages and then mapped other ASP.NET extensions to that handler as was appropriate:fastcgi.map-extensions = (
".asmx" => ".aspx",
".ashx" => ".aspx",
".asax" => ".aspx",
".ascx" => ".aspx",
".soap" => ".aspx",
".rem" => ".aspx",
".axd" => ".aspx",
".cs" => ".aspx",
".config" => ".aspx",
".dll" => ".aspx"
)
The base configuration for one of my virtual hosts (Urlenco.de) turned out something like this:
$HTTP["host"] == "urlenco.de" {
fastcgi.server = (
".aspx" => ((
"socket" => "/tmp/fastcgi-mono-server",
"bin-path" => "/usr/local/bin/fastcgi-mono-server2",
"bin-environment" => (
"MONO_FCGI_APPLICATIONS" => "/:/serv/www/domains/urlenco.de/htdocs",
"MONO_FCGI_LOGLEVELS" => "Standard", #All", #Debug",
"MONO_FCGI_LOGFILE" => "/var/log/lighttpd/mono.log",
),
"max-procs" => 4,
"check-local" => "disable"
))
)
}
Specifying the "application path" is somewhat of a pain, as now I more or less need a separate FastCGI configuration, which means they'll also need separate FastCGI servers, so another virtual host in the configuration (pineapple.monkeypox.org) has the following setup:
$HTTP["host"] == "pineapple.monkeypox.org" {
fastcgi.server = (
".aspx" => ((
"socket" => "/tmp/fastcgi-mono-server-pineapple",
"bin-path" => "/usr/local/bin/fastcgi-mono-server2",
"bin-environment" => (
"MONO_FCGI_APPLICATIONS" => "/:/serv/www/domains/pineapple.monkeypox.org/htdocs",
"MONO_FCGI_LOGLEVELS" => "Standard", #All", #Debug",
"MONO_FCGI_LOGFILE" => "/var/log/lighttpd/mono.log",
),
"max-procs" => 1,
"check-local" => "disable"
))
)
}
Since the virtual host pineapple.monkeypox.org barely runs any ASP.NET code at all, I decided to only give it one Mono FastCGI process. Also of note is that the "socket" is different from the other FastCGI handler, if you try to use the same socket, the first Mono FastCGI process will take it over and both FastCGI handlers will return the same code, returned from the first handler.
Feel free to bug me with any questions, this is my first foray into using Lighttpd and I'm already pleased as punch with it (compared to Apache) but there are certainly some caveats and bits of black magic involved with Mono, FastCGI and Lighttpd. That said, it still feels less sticky than running Apache 2 and mod_mono (not that they're not great and all). Hopefully web traffic will increase and give me a good test-bed for figuring out "the right stuff" to scale Mono on Lighttpd.
Scary thought isn't it? :)