When types and definitions aren't enough.

Was in #haskell on freenode a bit this morning, and someone mentioned something about how they were not exactly excited about the new rules for code formatting on if-then-else expressions.

I mentioned that I try to avoid if-then-else and case as much as possible by using types like Maybe that have 2 kinds of constructors, namely Nothing and "Just a" (for Maybe a).

I said that I can use the MonadPlus instance for Maybe a to get a lot of what is available in if-then-else clauses.

let x = someExpression

in if x == Nothing

then 9

else fromJust x

could be written as

let x = someExpression

in fromJust $ x `mplus` Just 9

mplus is defined for Maybe as evaluating the first parameter, and if it is not Nothing, it returns it, otherwise it will return the second parameter. It's essentially an "or" operator.

However, someone pointed out that there's absolutely no requirement for mplus to be written this way. It can still live up to all the rules and restrictions of MonadPlus by short-circuiting evaluation on the second argument instead of the first. Sure, it's sort of a de-facto first then second sequencing of evaluation, but it is not as safe as say "if-then-else".

I wonder now about the Applicative module as well, and specifically Alternative for the Maybe class.

I could just as easily write

let x = someExpression

in fromJust $ x <|> Just 9

But do we fall into the same trap of no guarantees? Is there a rule in Applicative enforcing the short-circuit of the first argument before the second?

Much code is written in the Applicative style for Parsec, so I really hope this is well defined.

What's Missing in the Haskell Community?

Documentation is often sighted as possibly the #1 item that needs to be improved with respect to Haskell. It depends on what modules we use, but I have to agree. It's quite difficult to uphold the claim that you don't need to understand Category Theory in order to employ a Monoid or Monad, but when you run into Monoid instances like "endo" and don't know what to make of it because the documents don't really describe how to use it, you're probably going to suffer in that that implementation of Monoid is likely useful for some kind of programming that you want to do, and you'll end up struggling with a solved problem. Some people have been stepping up to improve the documentation, and that's really wonderful, but I think there's still some work to be done there.

The best way to learn Haskell in general for me has been to get the great books that are available out there. Real World Haskell is freely available online. (But please support the authors and get a copy if you're finding it useful). Search for Haskell on Amazon.com and you'll find that the reviews are a really good guide to picking which ones might be right for you. If you're really new to the language. Dr. Graham Hutton's book is outstanding. There's even been a series on MSDN's channel 9 walking through the chapters of this book, explaining how to solve some problems and think like a functional programmer.

To keep up to date with Haskell developments, reddit has been invaluable. You'll find blog posts, updates about new Haskell packages, and general community news and related discussion topics there.

So what's still missing?

I can tell you that over the years I've been messing around with Haskell, trying to understand how it works, why it's appropriate for certain kinds of problem solving, and to really get a good appreciation of why people seem to really like it so much, that I feel like the community has been pretty amazing with respect to fueling the flames of curiosity.

Where I think we might be needing a little more help is in the following areas of Haskell.

Explaining where laziness or non-strict evaluation is an advantage over strict evaluation. Perhaps this requires learning to think about the code we write differently, in much the same way it can be a leap to get to recursive programming, I feel this might be a slightly wider gap to cross mentally. (But then again maybe I'm just getting old... )

Show more examples of unintended data-growth or space leakage due to the lack of strict evaluation. In languages like C, you're in direct control of when memory is allocated or deallocated. This is generally considered a "bad thing" for a lot of tasks, including systems programming if you are signed up with the Go camp. A side effect of non-strict by default seems to be that you have to understand how the code you're writing will be evaluated from a bit of a wider view than you might need to care about malloc and free, or new or delete. It seems that unless you've somehow been taught how to recognize the patterns that could cause a a space leak, you're basically doomed to run into some sharp corners that others already seem to understand how to avoid.

Real World Haskell has a great chapter on optimization, but perhaps it's time for an "Optimizing Haskell" book too? There's lots of good advice scattered all over the web, and the experts are not shy to offer you help should you ask. Sometimes I think it's difficult to even ask the right questions when you're confused though, and I suspect this may turn some folks off to Haskell.

L4 and Plan 9 or Inferno or both?

Some folks already started in on a Plan 9 port to L4 it seems (PDF), but I'm not sure how far they've gotten with it. I've been sort of peeping at L4 on and off for a long time now, and OKL4 has been successfully shipped in a few ARM phones, successfully being used to host Linux and a Qualcomm OS that drives the phone's radios and such. Neat stuff.

Been wondering if Plan 9 or Inferno and L4 are really a good marriage and what benefits could be added by leveraging Plan 9's namespace based resource management and L4's powerful IPC mechanisms.

As I've had a little more spare time lately, I've been digging around looking into L4 again, and I'm interested in exploring some ideas a little more deeply.

There's a few different implementations of L4 to look at...

OKL4 is now basically an ARM only platform in it's latest releases. Fiasco is a Pentium targeted L4 implementation that has a userland implementation that might be fun to work with (Fiasco-UX), and Pistachio is still being worked by at least a few people, with the latest changes coming in as of yesterday. Pistachio also supports a lot of architectures.

I'm tempted to play with each of these, but my problem has always been one of focus when it comes to these spare time projects as free time for me is usually at a premium.

Tim Newsham has been following the seL4 stuff, and OKL4 is migrating towards those APIs. His vote is I shoot for OKL4, so I believe that's where I'm going to start. There's a good community around that implementation, and a commercial pressure to keep things working nicely.

But as they say, talk is cheap... Let's see what I learn.

(UPDATE!)

Looks like a lot of progress was made by others who have already started this work!

Mouse editing... useful but not popular?

This is just a bit of a rant, possibly about something people think is really trivial to talk about but I think there's something worth saying here.

I've been noticing a lot of blogs and people using tools claiming that never taking one's hands off the keyboard is a good thing. This seems to be generally accepted as a rule of thumb to follow, but I don't buy it.

There's window managers that abolish the mouse completely for X such as Xmonad and ratpoison. They certainly have their merits and people are adopting them and finding that they get more work done than in alternative window managers. All of that is fine, I'm not trying to take anything away from those people or those projects, I'd just like to challenge the premise that hardly ever using the mouse is always optimal.

A few examples to the contrary that I've found for myself:

Expose in Mac OS X allows me to quickly use my touchpad, or trackball/mouse to point at a location on my desktop and quickly see all my windows and nearly immediately point to the window I'm interested in. To me this is a lot more useful than memorizing which of many desktops I might have decided to put certain windows on, or tabbing through to cycle to the correct window, or possibly remembering what shortcut key goes to a particular window on the current desktop. Point and click is easy.

Still I've peers who don't like Expose and do everything they can to turn it off.

When editing text, GUI versions of Emacs allow me to sweep over text and use that as a copy command to move text around during editing sessions. This is not as powerful as the Acme editor from Plan 9, in that I can chord buttons on my trackball to cut and paste text to different locations without taking my hand off the mouse.

I will admit I'm a bit of an emacs junkie, for better or worse, and like my elisp based customizations I've become used to over the years do save me a bit of time for certain tasks. Those are nearly always keyboard accessible shortcuts, but again, this becomes a memorization process of those commands. That memorization and customization is initially a distraction from getting work done, but it does become habit after a short time, and then it feels very fluid. I still often find myself switching between Acme, Sam, and emacs depending on the sort of editing I'm doing.

I don't see how selecting text with arrow keys or control key combinations is ever faster than a mouse sweep and click. In fact while editing this blog I find myself taking my hands off the keyboard, pointing at a word or phrase to double click and change a word very quickly, much faster than if I had to press repetitions of keys and delete buttons to change text.

I'll also go ahead and admit that before I'd used Acme, and I mean really used it for a little while, I was in the camp of people who would have said that never removing the hands from the keyboard was faster without even questioning why that is.

My advice to people is to try some of these other editors, and see if you can understand where I'm coming from. The people who designed the editors that are mouse driven have written large amounts of code in their day, and they've certainly gained a lot of experience in knowing what annoys them during an editing session and wanted to make changes to make the process more comfortable for themselves.

Worst case scenario is you'll still think that keyboard-only is the way to go and you'll not really have lost anything but maybe have re-assured yourself that for your work environment, that's the best way for you to go. Maybe you'll learn a new way to work that you find is faster.

Also, there's a lot more reasons to really like Sam or Acme, but that's for another time.

Unemployed

Well the company I work for is in the process of restructuring, and most everyone has been laid off. Verari Systems was a fun place to work, and I got to do a few of the things that I've nearly documented here (except for anything they'd consider Intellectual Property). It's been a fun ride, and there's rumors floating around about people wanting to re-invigorate our technology into a new company.

I'm hoping I can play a part in that new company if it happens, but I'm not currently willing to wager on that.

Started interviewing at one company, and I've been working a contract that'll get me through the holiday season, but I'm currently on the market if there's anyone interested!

I'll post updates to my employment status here as things develop.

I'd also like to say that the outpour of support I've gotten from friends has been amazing, and that I'm so glad to have people like them in my life.

Lazy Programming, too much, too little?

Another Haskeller/Blogger wrote the following interesting article on how optional laziness doesn't always work out so well.

However I'm finding there's a lot of functions that don't make a ton of sense to my imperative, strictly trained programmer's mind to do in a lazy way by default.

foldl is a good example. Without strictness checking turned on in the compiler options for GHC, it can turn into a memory leak right away.

I find myself kind of hard pressed to come up with a good use for a lazy foldl where it isn't obvious that I'd rather have used foldl'.

Long running Haskell applications

I've been using Haskell in a serious way for about 2 years. Been using it in a professional sense about 1.5 years now in that, yes, I am one of the lucky ones that gets to use Haskell at work.

The ride has been pretty smooth most of the time, as I've found that the type system especially helps me to rule out certain classes of bugs, easily test rather large chunks of programs as they're pure. The interpreter allows experimentation and iteration of ideas that can then be composed into the final compiled programs. All of this gives me a good deal of confidence that the code I'm writing is correct to some degree up front, something I've come to expect from functional programming languages over the years and greatly appreciate.

However I've felt compelled to comment that things aren't always so smooth either. I spent the better part of a weekend and a Monday tracking down a space leak in a program that just was not allowed to leak space. I have a stack of a ReaderT StateT IO that I use to communicate with a device through a passthrough program that speaks CAN to a device for the purposes of creating a serial console connection where there is only a CAN bus. The Haskell program is responsible for the management of the data found at the other end o the serial connection and supports operations to the device through the serial channel via a simple text protocol while "forever" polling data on the serial endpoint.

What I had done was the equivalent of

pollerLoop :: Poller ()
pollerLoop = forever pollOnce

Where Poller is my monad stack.
pollOnce is defined as,

pollOnce :: Poller ()
pollOnce  = do
checkCommandChannel -- see if there's a pending command to run
executePoll

Yes, this application is multi-threaded. I have a logger thread, a thread watching standard input of the program for queries and issuing commands to the endpoint. I have a thread per Poller, and the ability to poll devices simultaneously.

The poller includes an implementation of my little "expect" syntax which was based on a naive implementation of hGetChar and checking for a desired result or timing out eventually. The really data inefficient version is a real screamer, beating the heck out of my Parsec or even ReadP version, but because of the way I wrote it, with a lot of reversing and prefix checking and substring slicing into temporary areas, it's not useful for large input blocks over a long time frame.

Still it's so fast that it's appropriate for certain sections of code. I measured and verified this via space profiling to see the real runtime heap utilization (nice feature btw, I'd be dead without it right now I think).

So what's the problem you're probably thinking? Well it turns out that since part of my state in StateT is a Data.Map, and that all my polling and parsing of expect-passing blocks caused updates to a Map, coupled with the language's default laziness caused a bit of a "bomb" of PAP (Partial APplications of functions).

I had an ill-timed, project wise, discovery of a space leak.

I tried sprinkling $! and seq all over the place, rewriting big chunks of code that used Text.Regex.Posix, to use Parsec and only got incremental improvements. The growth problem still existed, and would eventually exhaust memory. This was a real problem as this was an application that was not supposed to stop when the other conditions of the management system were ok. It could run for months or even years!

I went through a whirlwind of emotions, and considered that perhaps I should be working a different job. Perhaps I could be a lion tamer? It turned out what I thought was a lion was really an anteater though... but that's literally a different story.

It turns out that by looping not inside the monad, but over the execStateT/runReaderT expression I could pull all the state out, and then re-inject it into another execStateT/runReaderT each poll, which forced the strictness I needed on all state data, made the system respond faster, and best of all, not crash!!!!

Diagrams below:

This first one is the "before" picture. It shows the data growth by cost center in my code. As you can see things are getting worse almost linearly as I poll.

This picture illustrates the result of pulling the state out of the Monad, and re-injecting it, forcing it to be evaluated. As you can see, I've got much more manageable memory utilization.

This final one shows the new algorithm running with 2 threads, one spawned a few seconds into the run. You can see the initial burst of the fast but inefficient "expect" algorithm, followed by a much more regular memory utilization pattern.

I'd like to thank all the folks on #haskell on FreeNode who gave me suggestions and hints to my vague problems regarding data growth and laziness vs strictness. Haskell's got a great user community and is probably one of the most helpful out there. One can learn a lot just asking questions of the right mentors as well as by reading haskell-cafe and the various blogs that are out there.

I'm hoping that this anecdote is useful to someone.