An Opinionated Survey of Functional Web Development
(Published on 2017-05-28.)
TL;DR: In this article I provide an overview of the frameworks and libraries available for creating websites in statically-typed functional programming languages.
I recommend you now skip directly to the next section, but if you're interested in some context and don't mind a rant, feel free to read on. :-)
When compared to native desktop application development, web development just sucks. Native development is relatively simple with toolkits such as Qt, GTK+ and others: You have convenient widget libraries, and you can describe your entire application, from interface design to all behavioural aspects, in a single programming language. You're also largely free to structure code in whichever way makes most sense. You can describe what a certain input field looks like, what happens when the user interacts with it and what will happen with the input data, all succinctly in a single file. There are even drag-and-drop UI builders to speed up development.
Web development is the exact opposite of that. There are several different technologies you're forced to work with even when creating the most mundane website, and there's a necessary but annoying split between code that runs on the server and code that runs in the browser. Creating a simple input field requires you to consider and maintain several ends:
- The back end (server-side code) that describes how the input field interacts with the database.
- Some CSS to describe what the input field looks like.
- And then there's HTML to act as a glue between the above.
In many web development setups, all four of the above technologies are maintained in different files. If you want to add, remove or modify an input field, or just about anything else on a page, you'll be editing at least four different files with different syntax and meaning. I don't know how other developers deal with this, but the only way I've been able to keep these places synchronized is to just edit one or two places, test if it works in a browser, and then edit the other places accordingly to fix whatever issues I find. This doesn't always work well: I don't get a warning if I remove an HTML element somewhere and forget to also remove the associated CSS. Heck, in larger projects I can't even tell whether it's safe to remove or edit a certain line of CSS because I have no way to know for sure that it's not still being used elsewhere. Perhaps this particular case can be solved with proper organization and discipline, but similar problems exist with the other technologies.
Yet despite that, why do I still create websites in my free time? Because it is the only environment with high portability and low friction - after all, pretty much anyone can browse the web. I would not have been able to create a useful "Visual Novel Database" any other way than through a website. And the entire purpose of Manned.org is to provide quick access to man pages from anywhere, which is not easily possible with native applications.
I'm confident that what I describe is possible, and it's evident that I'm not the only person to want this, as several (potential) solutions like this do indeed exist. I've been looking around for these solutions and have experimented with a few that looked promising. This article provides an overview of what I have found so far.
My adventure began with OCaml. It's been a few years since I last used OCaml for anything, but development on the language and its ecosystem have all but halted. Real World OCaml has been a great resource to get me up to speed again.
The framework comes with an embedded DSL with which you can conveniently generate HTML without actually typing HTML. And best of all, this DSL works on both the client and the server: On the server side it generates an HTML string that can be sent to the client, and running the same code on the client side will result in a DOM element that is ready to be used.
Ocsigen makes heavy use of the OCaml type system to statically guarantee the correctness of various aspects of the application. The HTML DSL ensures not only that the generated HTML well-formed, but also prevents you from incorrectly nesting certain elements and using the wrong attributes on the wrong elements. Similarly, an HTML element generated on the server side can be referenced from client side code without having to manually assign a unique ID to the element. This prevents accidental typos in the ID naming and guarantees that the element that the client side code refers to actually exists. URL routing and links to internal pages are also checked at compile time.
Ocsigen almost exactly matches what I previously described as the perfect development framework. Unfortunately, it has a few drawbacks:
- The framework has a steep learning curve, and the available documentation is by far not complete enough to help you. I've found myself wondering many times how I was supposed to use a certain API and have had to look for example code for enlightenment. At some point I ended up just reading the source code instead of going for the documentation. What doesn't help here is that, because of the heavy use of the type system to ensure code correctness, most of the function signatures are far from intuitive and are sometimes very hard to interpret. This problem is made even worse with the generally unhelpful error messages from the compiler. (A few months with Rust and its excellent error messages has really spoiled me on this aspect, I suppose).
- I believe they went a bit too far with the compile-time verification of certain correctness properties. Apart from making the framework harder to learn, it also increases the verbosity of the code and removes a lot of flexibility. For instance, in order for internal links to be checked, you have to declare your URLs (or services, as they call it) somewhere central such that the view part of your application can access it. Then elsewhere you have to register a handler to that service. This adds boilerplate and enforces a certain code structure. And the gain of all this is, in my opinion, pretty small: In the 15 years that I have been building web sites, I don't remember a single occurrence where I mistyped the URL in an internal link. I do suppose that this feature makes it easy to change URLs without causing breakage, but there is a trivial counter-argument to that: Cool URIs don't change. (Also, somewhat ironically, I have found more dead internal links on the Ocsigen website than on any other site I have visited in the past year, so perhaps this was indeed a problem they considered worth fixing. Too bad it didn't seem to work out so well for them).
Despite these drawbacks, I am really impressed with what the Ocsigen project has achieved, and it has set a high bar for the future frameworks that I will be considering.
I have always seen Haskell as that potentially awesome language that I just can't seem to wrap my head around, despite several attempts in the past to learn it. Apparently the only thing I was missing in those attempts was a proper goal: When I finally started playing around with some web frameworks I actually managed to get productive in Haskell with relative ease. What also helped me this time was a practical introductory Haskell reference, What I Wish I Knew When Learning Haskell, in addition to the more theoretical Learn You A Haskell for Great Good.
Haskell itself already has a few advantages when compared to OCaml: For one, it has a larger ecosystem, so for any task you can think of there is probably already at least one existing library. As an example, I was unable to find an actively maintained SQL DSL for OCaml, while there are several available for Haskell. Another advantage that I found were the much more friendly and detailed error messages generated by the Haskell compiler, GHC. In terms of build systems, Haskell has standardized on Cabal, which works alright most of the time. Packaging is still often complex and messy, but it's certainly improving as Stack is gaining more widespread adoption. Finally, I feel that the Haskell syntax is slightly less verbose, and more easily lends itself to convenient DSLs.
Despite Haskell's larger web development community, I could not find a single complete and integrated client/server development framework such as Ocsigen. Instead, there are a whole bunch of different projects focussing on either the back end or the front end. I'll explore some of them with the idea that, perhaps, it's possible to mix and match different libraries and frameworks in order to get the perfect development environment. And indeed, this seems to be a common approach in many Haskell projects.
Let's start with a few back end frameworks.
Scotty is a web framework inspired by Sinatra. I have no experience with (web) development in Ruby and have never used Sinatra, but it has some similarities to what I have been using for a long time: TUWF.
Scotty is a very minimalist framework; It does routing (that is, mapping URLs to Haskell functions), it has some functions to access request data and some functions to create and modify a response. That's it. No database handling, session management, HTML generation, form handling or other niceties. But that's alright, because there are many generic libraries to help you out there.
Thanks to its minimalism, I found Scotty to be very easy to learn and get used to. Even as a Haskell newbie I had a simple website running within a day. The documentation is appropriate, but the idiomatic way of combining Scotty with other libraries is through the use of Monad Transformers, and a few more examples in this area would certainly have helped.
Continuing with the Star Trek franchise, there's Spock. Spock is very similar to Scotty, but comes with type-safe routing and various other goodies such as session and state management, CSRF protection and database helpers.
As with everything that is (supposedly) more convenient, it also comes with a slightly steeper learning curve. I haven't, for example, figured out yet how to do regular expression based routing. I don't even know if that's still possible in the latest version - the documentation isn't very clear. Likewise, it's unclear to me what the session handling does exactly (Does it store something? And where? Is there a timeout?) and how that interacts with CSRF protection. Spock seems useful, but requires more than just a cursory glance.
Servant is another minimalist web framework, although it is primarily designed for creating RESTful APIs.
Servant distinguishes itself from Scotty and Spock by not only featuring type-safe routing, it furthermore allows you to describe your complete public API as a type, and get strongly typed responses for free. This also enables support for automatically generated documentation and client-side API wrappers.
Servant would be an excellent back end for a SPA, but it does not seem like an obvious approach to building regular websites.
- Happstack / Snap / Yesod
Happstack, Yesod and Snap are three large frameworks with many auxiliary libraries. They all come with a core web server, routing, state and database management. Many of the libraries are not specific to the framework and can be used together with other frameworks. I won't go into a detailed comparison between the three projects because I have no personal experience with any of them, and fortunately someone else already wrote a comparison in 2012 - though I don't know how accurate that still is today.
So there are a fair amount of frameworks to choose from, and they can all work together with other libraries to implement additional functions. Apart from the framework, another important aspect of web development is how you generate the HTML to send to the client. In true Haskell style, there are several answers.
For those who prefer embedded DSLs, there are xhtml, BlazeHTML and Lucid. The xhtml package is not being used much nowadays and has been superseded by BlazeHTML, which is both faster and offers a more readable DSL using Haskell's do-notation. Lucid is heavily inspired by Blaze, and attempts to fix several of its shortcomings. Having used Lucid a bit myself, I can attest that it is easy to get started with and pretty convenient in use.
Fay itself doesn't come with many convenient UI libraries, but Cinder covers that with a convenient HTML DSL and DOM manipulation library.
Fay is still seeing sporadic development activity, but there is not much of a lively community around it. Most people have moved on to other solutions.
The community around GHCJS seems to be more active than that of Fay. GHCJS integrates properly with the Stack package manager, and there are a whole bunch of libraries available.
While it should be possible to share a fair amount of code between the front and back ends, not all libraries work well with Haste. I tried to use Lucid within a Haste application, for example, but that did not work. Apparently one of its dependencies (probably the UTF-8 codec, as far as I could debug the problem) performs some low-level performance optimizations that are incompatible with Haste.
Haste itself is still being sporadically developed, but not active enough to be called alive. The compiler lags behind on the GHC version, and the upcoming 0.6 version has stayed unreleased and in limbo state for at least 4 months on the git repository. The community around Haste is in a similar state. Various libraries do exist, such as Shade (HTML DSL, Reactive UI), Perch (another HTML DSL), haste-markup (yet another HTML DSL) and haste-dome (yet another HTML DSL), but they're all pretty much dead.
Despite having three options available, only Haste provides enough benefit of code reuse while remaining efficient enough for the kind of site that I envision. Haste really deserves more love than it is currently getting.
In my quest for Haskell web development frameworks and tools, I came across a few other interesting libraries. One of them is Clay, a CSS preprocessor as a DSL. This will by itself not solve the CSS synchronisation problem that I mentioned at the start of this article, but it could still be used to keep the CSS closer to code implementing the rest of the site.
I've covered OCaml and Haskell now, but there are relevant projects in other languages, too:
The PureScript community is very active and many libraries are available in the Persuit package repository. Of note is Halogen, a high-level reactive UI library. One thing to be aware of is that not all libraries are written with space efficiency as their highest priority, the simple Halogen button example already compiles down to a hefty 300 KB for me.
Ur/Web is an ML and Haskell inspired programming language specifically designed for client/server programming. Based on its description, Ur/Web is exactly the kind of thing I'm looking for: It uses a single language for the front and back ends and provides convenient methods for communication between the two.
This has been a low priority on my to-try list because it seems to be primarily a one-man effort, and the ecosystem around it is pretty small. Using Ur/Web for practical applications will likely involve writing your own libraries or wrappers for many common tasks, such as for image manipulation or advanced text processing. Nonetheless, I definitely should be giving this a try sometime.
(Besides, who still uses frames in this day and age? :-)
I'll be moving out of the functional programming world for a bit.
In the same vein, there's Wt. The name might suggest that it is a web-based clone of Qt, and indeed that's what it looks like. Wt is written in C++, but there are wrappers for other languages. None of the languages really interest me much, however.
That said, if I had to write a web UI for a resource-constrained device, this seems like an excellent project to consider.
To be honest, I am a bit overwhelmed at the number of options. On the one hand, it makes me very happy to see that a lot is happening in this world, and that alternatives to boring web frameworks do exist. Yet after all this research I still have no clue what I should use to develop my next website. I do like the mix and match culture of Haskell, which has the potential to form a development environment entirely to my own taste and with my own chosen trade-offs. On the other hand, the client-side Haskell solutions are simply too immature and integration with the back end frameworks is almost nonexistent.
Almost none of the frameworks I discussed attempt to tackle the CSS problem that I mentioned in the introduction, so there is clearly room for more research in this area.
There are a few technologies that I should spend more time on to familiarize myself with. Ur/Web is an obvious candidate here, but perhaps it is possible to create a Haskell interface to Wt. Or maybe some enhancements to the Haste ecosystem could be enough to make that a workable solution instead.