Windows fun with Dart FFI
Native Win32 API support for Flutter and Dart
As a product manager for a developer framework and programming language, it’s not always easy to find time during the workday to write code. But I consider it a vital task in order to empathize with my customers’ needs. So I dabble here and there with various projects that pique my interest; and over the last couple of months, I’ve been exploring a project that combines my many years of working on Windows with my current focus on Flutter and Dart, culminating in a package that wraps a good portion of the Windows API for consumption from Dart and Flutter apps. But the journey itself is also quite a fun story.
Small steps: Console APIs
This all started with a small text editor. Via HackerNews, I came across Kilo, a UNIX-style terminal text editor written in less than 1,000 lines of C, and a very well-written tutorial that you can follow along to build it from scratch. I decided to give it a go, but porting the code to Dart as I went. This was a ton of fun.
What was not so fun? VT escape sequences, which is how Kilo manipulates the console for things like cursor movement, clearing the screen and hiding the cursor during screen updates. This archaic part of the modern Mac and Linux terminal has somehow survived for over forty years without significant modernization. In a remarkable piece of retro-refitting, it has even been introduced in recent years to the Windows terminal. At some point in the process, I forget when, I started factoring out some of the VT console commands into a separate package, so I could write something coherent like console.hideCursor() instead of stdout.write('\x1b[?25l').
After completing the tutorial, I had dart_kilo, a simple text editor that ran on macOS or Linux, just like the original, but in only ~500 lines of code, thanks to my separate lightweight console library.
But then I started wondering — could I port my Dart version of kilo to Windows? Most of the code worked fine, at least on a modern Windows terminal, with the exception of a couple of POSIX system calls to get the console window dimensions and set the terminal into raw mode, both of which would need conversion.
Introducing FFI
Dart includes dart:ffi, a library for making Foreign Function Interface calls to C-style APIs. Using FFI, you can declare a prototype for a C-based API and call it from your Dart code. As an example, the Win32 API function SetConsoleCursorPosition can be called with the following lines of Dart code:
Now all I had to do was to create Win32 equivalents of those POSIX system calls, and the kilo editor ran on my Windows laptop without a single change to the editor itself:
From the console to a graphical UI
For a while, I continued exploring the Win32 console APIs, and gradually wrapping more of them in my dart_console package. Dart has a relatively basic set of console functions, and so I gradually built up a reasonably feature-rich package that offered color selection, cursor manipulation, REPL-style command input and control key processing. By separating interface and implementation, I was able to add features with support on both Windows and UNIX-like terminals, even including the Windows 7 console app, which doesn’t support VT escape sequences.
But I was curious to see whether I could go further. Any self-respecting language should be able to call the Win32 MessageBox function, after all:
And then I started thinking about the sine qua non of Windows applications, the hello.c Hello World program presented by Charles Petzold in his seminal Programming Windows title. Could this be brought to Dart without needing Visual Studio or the Windows SDK?
hello.c is on the surface a much more complex challenge: it requires allocation of structs on the heap, callback functions, a WinMain() entry, and about twenty Win32 calls. At the time, I still hadn’t really got my head around structs in dart:ffi, nor fully grokked the translation between various Dart primitives and their C equivalents. In short, I wasn’t expecting success.
So after a fair amount of trial and error, I was somewhat stunned when Windows displayed this small but exciting window:
But what is cool is how similar the Dart code is to the original for anyone who cut their teeth writing traditional Windows code, albeit with the advantages of a somewhat more forgiving, strongly-typed language backing it:
From here, I started to get a lot more excited about the potential of this. I created a separate package for the Win32 APIs, and started to wrap more of them. I started looking for other small Windows C applications that were permissively licensed and porting them across to Dart.
For example, here’s Tetris:
And (thanks again to Charles Petzold), a Notepad implementation, including menus, shortcut keys, find/replace and font selection:
COM and Dart
By now I was getting more confident. In building (or translating) these more advanced apps with Dart, I’d wrapped a hundred or more Win32 APIs, including some more painstaking work to bring across the various constants and structs needed to use them, adding as many tests as I could, and starting to build out documentation for the embryonic package that I had released for pub.dev, the Dart package manager.
But I was starting to hit some limitations: in particular, more recent Win32 APIs often used the class-based COM model, which presented a different set of challenges. I figured this might be my glass ceiling, since the C-based interop libraries in Dart don’t mesh well with the C++ assumptions of COM. But then I discovered a lengthy CodeProject article dating back to 2006 that described how COM components could be called from plain old C, and so I decided to give it a go.
COM from C is ugly. A COM component supports one or more interfaces, such as IUnknown or IFileDialog, through which they expose methods to a calling application. The addresses of the methods themselves are stored in a virtual function table, and the details of the objects are stored in a rather painful format called MIDL (or Microsoft Interface Definition Language).
I used to know some things about COM, when I was a younger man, but by now all the brain cells holding this knowledge had long atrophied. All the good books on COM were written twenty years ago. (They advertise as a point of pride that they come with a CD-ROM, since the Internet was still far from pervasive at the time.) To my amusement, I found myself scavenging online used bookstores to try and recover replacements for titles I’d given away at the turn of the century. If you’d told me in 2000 that I’d find these books relevant in 2020, I’d have either laughed or wept.
I spent months — I mean months — trying to get the damn thing working. In theory I understood what I needed to do: initialize COM and create an instance of the class, dig into the vtable to find the method I needed, map the address of that method to the Dart prototype I’d created, and then call the result. But my commits of the time demonstrate a litany of failed attempts, with endless print statements and futile explorations in both Dart and C to try and figure out what I’d done wrong. I’d put it away for a couple of weeks and work on something else, and then come back to make the same mistakes I’d already made in the hope of a different answer. Dear reader: my coding is the equivalent of a random walk through a maze, hoping against hope that my Brownian motion will eventually lead me to the exit.
And then, suddenly, daylight! I found a pointer dereferencing bug for the third time, but on this occasion, the other two bugs that had prevented me making headway had been overcome. My code was horrid, but functional for the single method I needed.
One problem remained: the manual approach I’d adopted for Win32 was just unfeasible for COM. The IFileDialog interface alone is 23 methods, not counting the 30+ other methods I’d need to implement to implement the other associated interfaces. Clearly I needed a different approach. So I started a brute force parser for some of the header metadata in the Windows SDK, allowing me to read in a file like this:
and convert it to something like this:
This may be some of the least attractive Dart code ever written (at least, aesthetically), but that’s OK. It’s machine-generated, faithful to the original COM interface, and is unlikely to need close inspection. With a lightweight wrapper to shield the non-idiomatic code from the light of day, you can now write something like this:
Now I could start to build out other packages that depended on the COM APIs, such as filepicker_windows. And even write simple Windows utilities that combine Flutter UI with Win32 APIs:
The Win32 Package
Over the last few months, I’ve been gradually refining and improving the package, building out samples and adding documentation. The package now supports hundreds of APIs, and a wide variety of COM APIs, with code generators doing much of the heavy lifting. More recently, I’ve been working to provide a projection for the latest Windows Runtime APIs as used in UWP apps, which opens up some other intriguing possibilities. But that’s a story for another time.
One of the most gratifying aspects of contributing a package like this back to the community is having others file issues, depend on your code, or even submit pull requests. I’ve enjoyed seeing folk like Tomek Polanski use it for his fast_flutter_driver test harness, and working with others to add new APIs for packages like biometric_storage. We’re even now using it for the Windows implementation of Flutter packages like path_provider.
And as a product manager, this whole experience has continued to refine and improve my understanding of how our product feels to others. There are friction points that were previously merely academic and are now viscerally felt; I’ve found some new bugs (and filed issues for them); and submitted numerous documentation pull requests that hopefully ease the path for those following. At the same time, product managers need to avoid the highly seductive trap of putting too much emphasis on our own direct observations— we’re building for customers, and it’s critical to temper personal experience with data-driven insights and customer feedback.
Win32 is now available at pub.dev, and I’d be honored if you use it for your own projects.
