Search code examples
boonemerle

When is a new language the right tool for the job?


For a long time I've been trying different languages to find the feature-set I want and I've not been able to find it. I have languages that fit decently for various projects of mine, but I've come up with an intersection of these languages that will allow me to do 99.9% of my projects in a single language. I want the following:

  • Built on top of .NET or has a .NET implementation
  • Has few dependencies on the .NET runtime both at compile-time and runtime (this is important since one of the major use cases is in embedded development where the .NET runtime is completely custom)
  • Has a compiler that is 100% .NET code with no unmanaged dependencies
  • Supports arbitrary expression nesting (see below)
  • Supports custom operator definitions
  • Supports type inference
  • Optimizes tail calls
  • Has explicit immutable/mutable definitions (nicety -- I've come to love this but can live without it)
  • Supports real macros for strong metaprogramming (absolute must-have)

The primary two languages I've been working with are Boo and Nemerle, but I've also played around with F#.

Main complaints against Nemerle: The compiler has horrid error reporting, the implementation is buggy as hell (compiler and libraries), the macros can only be applied inside a function or as attributes, and it's fairly heavy dependency-wise (although not enough that it's a dealbreaker).
Main complaints against Boo: No arbitrary expression nesting (dealbreaker), macros are difficult to write, no custom operator definition (potential dealbreaker).
Main complaints against F#: Ugly syntax, hard to understand metaprogramming, non-free license (epic dealbreaker).

So the more I think about it, the more I think about developing my own language.

Pros:

  • Get the exact syntax I want
  • Get a turnaround time that will be a good deal faster; difficult to quantify, but I wouldn't be surprised to see 1.5x developer productivity, especially due to the test infrastructures this can enable for certain projects
  • I can easily add custom functionality to the compiler to play nicely with my runtime
  • I get something that is designed and works exactly the way I want -- as much as this sounds like NIH, this will make my life easier

Cons:

  • Unless it can get popularity, I will be stuck with the burden of maintenance. I know I can at least get the Nemerle people over, since I think everyone wants something more professional, but it takes a village.
  • Due to the first con, I'm wary of using it in a professional setting. That said, I'm already using Nemerle and using my own custom modified compiler since they're not maintaining it well at all.
  • If it doesn't gain popularity, finding developers will be much more difficult, to an extent that Paul Graham might not even condone.

So based on all of this, what's the general consensus -- is this a good idea or a bad idea? And perhaps more helpfully, have I missed any big pros or cons?

Edit: Forgot to add the nesting example -- here's a case in Nemerle:

def foo = 
    if(bar == 5)
        match(baz) { | "foo" => 1 | _ => 0 }
    else bar;

Edit #2: Figured it wouldn't hurt to give an example of the type of code that will be converted to this language if it's to exist (S. Lott's answer alone may be enough to scare me away from doing it). The code makes heavy use of custom syntax (opcode, :=, quoteblock, etc), expression nesting, etc. You can check a good example out here: here.


Solution

  • Sadly, there's no metrics or stories around failed languages. Just successful languages. Clearly, the failures outnumber the successes.

    What do I base this on? Two common experiences.

    1. Once or twice a year, I have to endure a pitch for a product/language/tool/framework that will Absolutely Change Everything. My answer has been constant for the last 20 or so years. Show me someone who needs support and my company will support them. And that's that. Never hear from them again. Let's say I've heard 25 of these.

    2. Once or twice each year, I have to work with a customer who has orphaned technology. At some point in the past, some clever programming built a tool/framework/library/package that was used internally for several projects. Then that programmer left. No one else can figure that darn thing out, and they want us to replace/rewrite it. Sadly, we can't figure it out either, and our proposal is to rewrite from scratch. And they complain that their genius built the set of apps in a period of weeks, it can't take us months to rewrite them in Java/Python/VB/C#. Let's say I've written 25 or so of these kinds of proposals.

    That's just me, one consultant.

    Indeed one particularly sad situation was a company who's entire IT software portfolio was written by one clever guy with a private language and tools. He hadn't left, but he'd realized that his language and toolset had fallen way behind the times -- the state of the art had moved on, and he hadn't.

    And the move was -- of course -- in an unexpected direction. His language and tools were okay, but the world had started to adopt relational databases, and he had absolutely no way to upgrade his junk to move away from flat files. It was something he had not foreseen. Indeed, it was something he could not possibly foresee. [You won't fall into this trap, will you?]

    So, we talked. He rewrote a lot of the applications in Plain-Old VAX Fortran (yes, this is a long time ago.) And he rewrote it to use plain old relational SQL stuff (Ingres, at the time.)

    After a year of coding, they were having performance problems. They called me back to review all the great stuff they'd done in replacing the home-built language. Sadly, they'd done the worst possible relational database design. Worst possible. They'd taken their file copies, merges, sorts, and what-not, and implemented each low-level file system operation using SQL, duplicating database rows left, right and center.

    He was so mired in his private vision of the perfect language, that he couldn't adapt to a relatively common, pervasive new technology.