My father decided to get a whole bunch of new furniture made (like we all do at some point in our lives). New beds, doors, chairs, closets. A carpenter was hired. Many kinds of woods were looked at and some were picked. I was tasked with tracking finances for the whole exercise.
I dropped into a Google Sheet and got started. By the end I had half a dozen worksheets, a few pivot tables and a variety of formulas. The work finished over a couple of weeks. We paid out the carpenter from the calculations in the sheet.
As a software engineer, I like to write and appreciate good software (no, I pine for it) – easy to work with, easy to understand, amenable to change, and reliable. The sheet was none of these things.
I was compelled to take a close look at why it was so. It’s been a few months since. This is a quick summary of things I’ve found, some of my thoughts and what I’ve been up to with them.
If you’ve spent enough time on a sheet you’ve seen these: copy-paste errors, finicky date formats, a mixup with a relative/absolute reference, some rows at the bottom missing from an average1, awkward VLOOKUPs, INDEX-MATCHes, workarounds for circular references2 and mega-formulas3. Put a few of these in a sheet and it gets messy very quickly.
I ran into some of the same problems
I see similar problems when programming, except there’s usually a way to solve them there.
Spreadsheet usage falls in any intersection of Storage, Presentation and Computation needs.
People don’t always do all three. Spreadsheets are predominantly used for storage and presentation only – to make lists and lay out tables (todo lists, project management trackers, etc). Most of them don’t contain any formulas6. However for the (small group of) users whose work falls in the computation circle they’re uniquely powerful. I wouldn’t plan this work anywhere else. Nothing else fits. Nardi & Miller7 ascribe this to
Here are some sheets similar to mine. A small company’s book of employees & salaries 8, a personal stock portfolio 9, a scientist organising some statistics 10, a crypto mining profit tracker 11.
These sheets are like regular programs in many ways. Felienne Hermans, a veteran spreadsheet researcher, puts it very simply: Spreadsheets are code12. She also goes on to show that they suffer from the same problems as real software.
❖
Simplistically, information systems are a place to put some information, do transformations on it (optionally) and look it up later13. A cash register, a cab booking app, a payment gateway, a search engine are all information systems in that sense. These sheets particularly are small information systems. I want to call them small software. The ask is dependable, good quality programs except their size – in surface area and complexity – is small.
Other names have been used synonymously (personal software, organizational software, end-user programs) but to me small captures the vibe well14. I also use small to distinguish from big software
We wanted tools to make sheds, we got tools to make skyscrapers.
–– a tweet I cannot find anymore
If we posit that spreadsheets are small software, how do they fare on the qualities we’re looking for? Consider the following
Over the years, developers have arrived at some must-haves to build good software. We like to use good languages, write tests, peer review our code etc. Very little of this has made its way to spreadsheets so it’s not surprising they’re
One, I think the spreadsheet medium is nascent, by the measure of how much has and hasn’t been tried out. There are many bad primitives and very few guardrails. We don’t get a new spreadsheet every year. We do this with programming languages all the time.
New spreadsheets or spin-offs that try to solve these problems usually depart from the recipe in some significant way. Experimental ones and dataflow programming have their own place but they are all something different. And it’s not reasonable to expect mainstream spreadsheets to solve this.
Secondly, I think patching your favorite programming language on top does not work well. Each language has its abstractions and ways of working, which may be at odds with the spreadsheet paradigm. Translating between paradigms has a real cost that a user has to bear. Users also realize that there are too many languages to deal with already and more of them is a problem17.
Thirdly, I think more people should be implementing their own spreadsheets. These things are probably not showing up in popular ones anytime soon
So we’re writing a spreadsheet at nilenso to try out some of these ideas. It’s intended to be a playground to implement things explored in theory and run experiments. Maybe eventually, it’ll become a full-fledged piece of software that I wouldn’t mind using. The plan is to write about the proceedings as we go.
Lastly, any other way of writing sane and functional programs requires a big commitment to learn and get started. The choice is between the scrappy utilities in existing spreadsheets and installing Python on your computer. I think spreadsheets still hold the promise of a well-formed environment for making small software; where the medium, the language, and the tooling works as one.
Thanks to Atharva, Prathyush and friends for reviewing this post.
Spreadsheet programming problems – leancrew.com (Blog, 2013) ↩
WHY and HOW to stop using circular references to calculate interest – Diarmuid Early (Video, 2022) ↩
Megaformula Examples – flylib.com (Blog) ↩
I solved this by referring to all rows till the bottom of the sheet in formulas. ↩
The solution here was a VLOOKUP or an INDEX-MATCH formula. ↩
How Trello is different – Joel Spolsky (Blog, 2012) ↩
The spreadsheet interface: A basis for end-user programming – Bonnie A. Nardi, James R. Miller (Paper, 1990) ↩
Example from You Suck at Excel with Joel Spolsky (Video, 2015) ↩
Example from A Relational Spreadsheet (Blog, 2020) ↩
Example from Data processing in Excel for IB Biology (Video, 2016) ↩
Example from a spreadsheet I found on reddit. ↩
Pure Functional Programming in Excel • Felienne Hermans • GOTO 2016 (Video, 2016) ↩
Information system – britannica.com (Article) ↩
Prathyush tells me this sounds similar to Convival computing. ↩
Excel incorrectly assumes that the year 1900 is a leap year – learn.microsoft.com (Article) ↩
There’s something to be said about how the rise of the finance industry has shaped the evolution of spreadsheet software. I’ll not say much but leave this bit from a show I thoroughly enjoyed watching. ↩
Python in Excel Review: The Good, The Bad, & The Ugly! Chandoo (Video, 2023) ↩
It’s Freedom to Put Things Where My Mind Wants : Understanding and Improving the User Experience of Structuring Data in Spreadsheets – advait.org (Paper, 2022) ↩
Software discourse tends to be disproportionately focused on the latter. I find myself contributing to this skew as well. Perhaps it’s because tools and technologies are more concrete and easy to talk about. It’s also easy to get people worked up about it.
Yet tools and technologies only exist in service of the first category, which fundamentally capture the highest-value bits of a developer’s job—concept mapping. Or making analogies. Pick whichever phrase that makes the most sense to you. I shall present a short motivating example for what I mean by concept-mapping.
Suppose you were to write a Sudoku solver. Our first job would be to make an analogy—how do I turn this concept that humans brains understand (“solving sudoku”) into a representation that a different processing substrate (here, your computer) can “understand” and process?
There’s so many possibilities! Here’s three of them.
Each analogy lends itself to different problem solving approaches. The choice of analogy will affect a bunch of things like:
What I have left out is the enormous amount of analogies that our Sudoku solving analogy is building on top of. Our computing substrate, encodes binary on-off switches (which our hardware is great at representing) to base-2 numbers. These base-2 numbers are itself used to encode computational verbs such as “add”, “subtract” or “check if something is zero”. These primitives give rise to higher-level procedures like “sort a list” or “turn this language into the instruction set that this procedure is written in”. Our Sudoku solving analogy will likely be written in the language that is transitively analogous to a bunch of on-off switches, which mirror what humans understand as “solving a Sudoku puzzle”.
What’s interesting here is the fact that as we climb out of this pit of analogies, the lower level analogies matter less and less. Indeed, when we were enumerating our ideas to represent a Sudoku board, we weren’t thinking of on-off switches, base-2 numbers or even instruction encodings for said numbers. We could safely ignore them, and yet provide meaningful value and understanding of how to tackle this problem.
If you are willing to entertain yet another analogy, the meaninglessness of lower levels is rather eloquently described by Douglas Hofstadter:
Consider the day when, at age eight, I first heard the fourth étude of Chopin’s Opus 25 on my parents’ record player, and instantly fell in love with it. Now suppose that my mother had placed the needle in the groove a millisecond later. One thing for sure is that all the molecules in the room would have moved completely differently. If you had been one of those molecules, you would have had a wildly different life story. Thanks to that millisecond delay, you would have careened and bashed into completely different molecules in utterly different places, spun off in totally different directions, and on and on, ad infinitum. No matter which molecule you were in the room, your life story would have turned out unimaginably different. But would any of that have made an iota of difference to the life story of the kid listening to the music? No—not the teensiest, tiniest iota of difference. All that would have mattered was that Opus 25, number 4 got transmitted faithfully through the air, and that would most surely have happened. My life story would not have been changed in any way, shape, or form if my mother had put the needle down in the groove a millisecond earlier or later. Or a second earlier or later.
What I am trying to say is that with a good abstraction, it isn’t the substrate where the magic is at, it is instead in the interaction or “motions” around the analogy boundaries. In the same way, Chopin’s étude is fairly robust to changes in the instrument, record player or performer (assuming all the notes are hit correctly). These changes in the substrate would likely still stir Hofstadter’s heart.
If the contracts between the analogy we are making and the analogies we are building on top of remain the same, the substrate broadly does not matter. If we could represent graphs, lists and tables using a computing substrate powered by quantum superpositions, cosmic rays and alignment of chakras instead of base-2 numbers, our Sudoku solving analogies would still hold. Or at least that’s the promise that an effective Software developer provides—”my analogy can stand well despite the shifts in the analogies and substrates below it.”
Which is why my definition of what a software developer is rather flexible compared to most people. She need not be a desk-jobber who writes a code on a text editor—the substrate is not important—a good software developer uses effective techniques to come up with analogies to solve problems.
All this is not to say that our choices of technology does not matter—it only matters to the extent where it helps us make effective analogies. For example, I consider Clojure to be a more enjoyable programming language than Go, because I believe the former’s expressiveness helps me make more robust analogies. Then again, I also understand that there’s a lot more to analogy-making than the programming language I select. A hyperfocus on technologies (especially prevalent in junior developers) distracts from what I consider to be the business of making analogies.
That’s it, bye bye.
After an afternoon replete with good food, coconut juice and punishing tropical heat, we went to the venue at NIT Calicut. As we waited for the microphones to be set up for the keynote, I had a chat with the soon-retiring professor who helped conceive FOSSMeet. He helped revive this edition of the conference after a three-year gap, and also had a part in the Free Software revolution in Kerala, which saw schools and government departments adopt Free Software.
He told me how important it is for this conference to continue. Young people these days are great with technology, and there are plenty of platforms to talk about it. What we need more critically is a platform to uphold the ideals and values of Free or Swatantra Software. That is software that respects the four essential freedoms. I used to think like the Professor. But on that day, I didn’t have the heart to tell him what I really thought about the state of Free Software.
Free Software has already lost. It’s all about what helps corporations now.
I observed that many of the college-age attendees could not differentiate between Free Software and Open Source software. This was juxtaposed with an abundance of stickers and merchandise containing playful references to Richard M Stallman, the founder and larger-than-life figure that looms over the Free Software movement 1.
A lot of corrections and disambiguations were made over the next three days. With a few people prefixing their questions with an apology for still using Windows.
There were also some of the older speakers and Free Software supporters that made appeals to adopt and use Free and Open Source Software. Throw away the proprietary stuff run by large corporations who don’t have your best interests.
These ideals are great, I thought for a second. But this train of thought derailed soon.
Nearly all the great open-source alternatives being suggested here are fuelled and funded by the same giant corporations we are rallying against. That’s why we are getting to use it for free. GitHub is owned by Microsoft, and that’s where nearly all Open Source software lives.
I thought of the pride that some college professors and authorities would have if their students got placed in Microsoft or Google.
The talking points continued to be dispensed. Encourage freedom-respecting alternatives, even if they are slightly worse. The beauty of FOSS is that if something does not work for you, or if a feature is missing, we can always fix the software ourselves and share the changes back to help everyone else. My dissonance grew.
Yeah right. If the WiFi driver on someone’s laptop breaks, almost no one will spend their time learning kernel programming to figure out how to fix it. The same goes for most Free Software. People don’t have the expertise or time. The burden of ensuring that things work goes to thankless, burnt-out maintainers, or centralised entities (ie, big corporations). And what about the underpaid schoolteachers and officials in the rural districts running FOSS? Are we going to tell them to go fix everything for themselves?
The success stories of FOSS adoption in schools and governments were brought up.
Was FOSS adopted by the government merely to save costs? Do they even care about the four freedoms? Will they bother giving back to the community?
As the conference went on, my thoughts inner monologues got louder. I grew uncomfortable. Anyone who has read about mindfulness practices would know that denying reality causes stress and suffering.
We have our Free Software (and our Proprietary Software, and whatever software). Yet the world still ails from injustice, disease, poverty and suffering. If this conference is about something greater than cool technology, why aren’t we getting to the heart of the matter? What is the heart of the matter?
And on the morning of the second day, I felt we had suffered enough. I slanted over to fellow ensonians Prabhanshu2 and Akshatha and said, “Maybe there’s room for a talk full of bitter pills and harsh realities. It might offend, but it needs to be talked about.”
Quite wonderfully, my request for the world was willed into existence. The next speaker in line, Akshay S Dinesh, stepped up and supplied the bitter pills I was seeking, but he made them taste sweet with his fun, highly interactive talk, amusingly titled “Enough of JavaScript build tools. Solving real-world problems collaboratively through FOSS, etc”.
He laid out percentage figures with no descriptions in his slides and made people guess what these were. And systematically, the realisations started reverberating. The top 10% held around 70% of the nation’s wealth. If we owned a scooter, car or house, we were likely in that top 10%. He talked about catastrophic health expenditures and how a large chunk of Indians without our privileges are vulnerable to losing their wealth to medical emergencies. He talked about the prejudices that still exist in society and the mess we’re in.
Something changed for me in this Free Software conference, which until now was like many other such events, teeming with usual appeals to adopt Stallman-certified software. Now it finally felt like something more. We are finally talking about the big picture. Free Software might have tried to address this at some point. He laid bare what really ails us.
He left the audience with an exercise called “The 4th Box”3.
Each box displays the various attempts to fix injustices in our society. The first one is to apply the same solution equally, not accounting for the privileges people have. The other is something like affirmative action, where you solve each person’s needs differently, to get an equal outcome. The third box is to remove the obstacle entirely (why was it even there?). The fourth box is blank. It’s because there aren’t just three ways to address it. We need all of the good ideas we can get and more of them.
And right after he showed this, came the money line, which brought us back to Free and Open Source Software.
FOSS is an option.
It’s not the solution. It’s just an option, and not necessarily the best one. Use it with all the other tools in the box to help make the world more equitable.
Akshay did not talk about JavaScript.
I met some of the other speakers in the lunch break, mostly young whippersnappers like me who are all bundled up in a polythene bag labelled Gen-Z (usually by overly-confident journalists).
These were people like me, who were drawn to FOSS movements and had their fair share of contributions to various communities. And like me, they shared the same varietal of scepticism about FOSS. It was clear what the younger folks at the conference were focused on. We wanted to solve problems. We didn’t want to be activists tied to a cause. Use the tools we have to make the world better and get on with it.
There was the technical chatter too, of course. I was talking about TypeScript to the CTO of a company that runs a popular Open Source project. Types are a layer of restrictions added over JavaScript that guarantees the elimination of certain classes of errors. The debate we had was that while it’s useful to have these restrictions to protect fallible humans from themselves, we also eliminate a really large set of programs that are perfectly valid only because the type system doesn’t accept them. It’s a tradeoff.
I’d like to think about Free Software similarly. It’s a set of restrictions over the licensing and distribution of Software that if followed, will guarantee some good outcomes for society. It’s much harder to exploit your users when your program can be used, modified and inspected in pretty much any way. For example, if my local hospital’s patient management software met the Free Software definition, I would be able to verify if it did something nasty, like transmitting my prescription data to third parties. Moreover, a rural clinic would theoretically be able to take the same software and adapt it to make it work better in internet-scarce regions. Quite neat. But slapping a GPL licence on the patient management software would still not stop a malicious hospital from producing a data dump of all its data and selling it. They could still exploit their users if they really wanted to.
Stallman’s definitions of freedom will also rule out large classes of perfectly-good-for-society software with the villainous label of “unethical”. The Free Software Foundation is quite aggressive about things not meeting their standards. They see a binary, where there is a spectrum.
Stallman would lament the fact that my colleague Prabhanshu is working with MIT-licensed software at Simple. This license won’t meet the Free Software standards of “ethical” because someone could theoretically make modifications and deploy a proprietary variant that cannot be audited or trusted. While this is true, it does not change the fact that Simple is a software that has created highly positive outcomes for society. More so than a lot of software licensed in a way that would satisfy the Free Software maximalists.
And that’s why I am largely indifferent to Free Software as an ideology. Some say it’s dying, and others argue it’s already dead. I haven’t checked the pulse. I am not interested in another subculture that lives and dies by the holy words of its greatest founder.
The world is magical, but there are no magic spells. We don’t only need Free Software. We need something with a bigger frame, something more complete, and more of it.
Footnotes
]]>When someone told me about beanstalkd, the seed of an idea was planted in my mind. I had some free time on my hands and decided that I wanted to implement it in Clojure to learn a thing or two. The simplicity of the beanstalkd protocol appealed to me, and it seemed like something I could implement. In addition to this, a preview build of Project Loom was made available, and I wanted an opportunity to try out virtual threads. Over two months, this germ of an idea sprouted into the 2400 line sapling it is today. I referred only to the beanstalkd protocol, and didn’t read any of the beanstalkd code. Here’s how it works:
A job is a description of some task to be done. In beanstalkd, jobs are binary blobs. For simplicity, jobs are strings in small-stalk.
Here's how jobs are processed in a system using small-stalk:
There can be two types of clients of small-stalk: producers and workers. Producers submit jobs to be executed, and workers consume and execute these jobs. Note that small-stalk itself makes no distinction between clients, and any client can run any command.
small-stalk implements a subset of beanstalkd’s commands:
small-stalk’s acceptor thread listens on a port for incoming connections. When a connection is made, a connection thread is spawned to process commands sent by that client. The cheapness of virtual threads makes it possible to spawn a thread per connection, even if there are thousands of clients. Each connection thread reads commands, parses them and hands them off to the Queue Service for execution. The Queue Service uses the Persistence Service to persist the queue state to disk. The compactor thread regularly tells the Persistence Service to clean up data files on disk to save space.
Virtual threads are the JVM’s answer to goroutines and Erlang processes. They can be created much more cheaply than conventional OS threads, making it possible to create a thread per connection even if there are thousands and thousands of clients. Virtual threads are not available in a stable JDK yet, and I used a preview build of the JDK in order to try them out in small-stalk.
When a client connects to small-stalk, it is assigned an ID and its socket is stored in a registry. Identifying clients is necessary for some commands like reserve and delete, since if a job is reserved, only the client that reserved a job is allowed to delete it.
A virtual thread is spawned for every incoming connection. These connection threads take care of reading and writing data from and to the connection socket. The connection threads parse commands from text and hand them over to the queue service for execution. small-stalk does not support pipelining, which means that clients are expected to read the response of the previous command before sending a new command. Connection threads block until the Queue Service is done executing a command, because the Queue Service exposes a synchronous API.
The Queue Service is a combination of:
I knew that I wanted to use some kind of append-only log for persistence, since I thought this would be easier to implement than complex data structures on disk. This led me to the idea of serializing all data changes in a queue.
A mutation is a data representation of a change that must be made to the queue data structure. All mutating commands (i.e. commands that are not simple reads like peek) map to a mutation. There are also other mutations (such as ::time-to-run-expired) that are triggered by timers rather than by a connection thread.
All mutating commands made by connection threads are converted into mutations and enqueued into the mutation queue, to be later executed by a single mutation thread. This means that all state changes are serialised. While this may seem like an unnecessary performance bottleneck, it affords several benefits:
Since the Queue Service needs to expose a synchronous API, mutations are enqueued along with promises to which the mutation thread can deliver return values. reserve, in particular, needs to block until a ready job is available. To facilitate this, if a reserve mutation is processed and no job is available, the reserve mutation along with its return promise is enqueued into a separate queue for waiting reserves. When a new job is added, the mutation thread checks the waiting reserves queue and fulfils a waiting reserve, if any.
The reserve-with-timeout command offers a timeout on the reserve blocking. To implement this, a separate timer thread to cancel the reserve is launched. This thread does not mutate the state directly, but instead enqueues a special mutation indicating that the client’s timeout has expired. All state modification, including cleaning up and cancelling the reserve, is done by the mutation thread.
The time-to-reserve timeout works in a similar way. When enqueueing a job, the client can specify how long a worker should be allowed to keep a job reserved. After this timeout expires, small-stalk will automatically release the job. Again, a separate timer thread is launched when a client reserves a job. This thread enqueues a time-to-reserve-expired mutation, which is then handled by the mutation thread.
The Persistence Service is a collection of functions and data used to persist mutations processed by the Queue Service. Mutations are persisted before they are processed. The file format is a simple newline-separated list of EDN maps. Each map is a mutation. Mutations are appended to the file, one after another. This file is called an AOF (append-only file).
To prevent disk usage from getting out of hand, every so often, mutations in the AOF file are processed and replaced with a snapshot of the Queue Service’s state. This is done by the compactor thread. To facilitate this process (called checkpointing), mutations are actually split over multiple AOF files. One file can hold at most a configurable number of mutations. Once this limit is reached, a new file is created and new mutations are appended there. The compactor thread reads mutations from all but the newest file (to which mutations are currently being written), replays the mutations to build a state snapshot, then persists that snapshot to disk. Once this is done, the older AOF files are deleted.
To facilitate testing, the Persistence Service is stubbed out with a service that simply writes mutations to a string instead of the disk, and the compactor thread is not started.
In order to work with Clojure’s reference types, the queue data structure is immutable, implemented using a combination of Clojure’s immutable data structures.
Clojure offers an obscure but useful PersistentQueue data structure. However, this doesn’t support priorities. small-stalk’s Persistent Priority Queue is implemented as a map of priority numbers to PersistentQueues. Popping an element from the queue involves finding the smallest key, and popping an item from the corresponding PersistentQueue. Unless the number of different priorities grows to be very large (which is unlikely for most use cases), this is a fairly simple and efficient implementation. The map could be replaced by a sorted-map to make this even more efficient.
Separating the networking layer from the queueing system allowed me to easily unit test the Queue Service without having to send CRLF-terminated strings over a socket. Using a dependency injection framework like integrant also allowed me to construct and stub out stateful components as necessary in tests. Although I didn’t take the time to write end-to-end integration tests, those would be very helpful as well.
small-stalk is not without issues, which I haven’t had time to address yet.
A handful of recipes from the people at nilenso for your reading pleasure. These have all been tried by real cooking noobs and are hopefully simple enough for anyone to follow along. Having said that, if you do burn your cookers don’t @ us on twitter or instagram.
Break two eggs, add a bit of milk and sugar and mix it up. I use a fork for this typically. If you have chocolate sauce you could add it to this batter directly, cinnamon also. You'll dip the bread into the batter and then fry it on the pan, so make sure the bread fits into whatever it is you're using to make the batter in. I usually break the eggs into another pan or a walled plate. In my experience, two eggs is usually enough for six slices of bread, which serves two.
Then, put a pan on the gas. Add a tiny amount of oil, it's just to prevent sticking and prevent the butter from burning. Put a bit of butter on the pan. Then take a slice of bread, dip both sides into the batter, let the excess batter run off and then fry the bread. Be sure to do both sides. Repeat until you run out of batter. Try not to overcook the toast, I think it's nicer when soft.
You could eat the French toast with jam or something if you want, or you could make the apples like I'm about to tell you.
Slice up an apple into bite sized chunks. Lightly dust the bottom of a vessel with sugar and melt it. Be very careful, if you heat the sugar too long it'll burn, and if you let the sugar cool off it'll stick to the vessel and become a real pain in the ass. No pressure. Add the apples in at this point along with some butter, mix it up nicely. I've also added honey in at this point as well, that's quite nice. Then add a bit of water. The apples should be partly submerged. Let them simmer on the stove. You want the water to evaporate.
You'll be left with the apples and some very sweet sauce as well. Usually I prepare the apples in the vessel, let them simmer and then start with the toast. They'll get done together.
Put the apples and sauce over the French toast and enjoy.
“Sandy's french toasts are p yum. Especially when it's already made and left on the kitchen counter by him for you to eat.”
“Fantastic. The apples were very surprising.”
The whole exercise takes around 12 minutes.
“Simple, high ROI recipe that will make you feel like ten times the cook you actually are. The controversy generated by the raisins cements this recipe's place as a work of art.”
“Used the wrong dal, the wrong red chilli and had no idea what consistency to expect but delicious. Nifty use of dry stores. 10/10 kajukishmish”
Heat the oil, add mustard and curry leaves. After a few seconds add onion and saute till transparent. Add chillies and tomatoes. Close the lid and let it cook on high flame.
After 3 min smash and saute it until the tomatoes are fully cooked. Now add the chilli powder, turmeric powder and salt. Mix it well and add some water. Cook until it boils.
“I over-improvised a perfectly self-contained recipe and regretted the results. Looking forward to trying again.”
This one has no real value add over reading the back of the packet of oats you buy and Milo is not available in stores anymore, but here it is irregardlessly:
“It was pretty decent I would say.”
(misrepresentation)
“Couldn’t discern the texture, overcooked it and made regular scrambled eggs, but it turned out damn delicious. Best thing about trying to get this right is that the baseline is pretty delicious.”
“Am I going to hype up these scrambled eggs? I guess I am. Let me tell you, if you nail this thing, you are going to have the best scrambled eggs in your whole life. Possibly the best egg anything in your whole life. It’s really that good.”
You’ll know it’s done once you get this thick, creamy, chunky texture. And stop the heat before you think it’s ready. The egg will continue to cook in its heat. I only learned too late about the cornstarch shortcut. The main concept is that you gotta let your eggs cook really slow. Basically, torture the eggs and make them almost solidify but not quite every time.
Eat it with some mildly toasted rustic bread. Wheat bread also works. If you are feeling fancy a thick hunk of sourdough would be perfect.
Let me set the scene. A nurse named Ravdeep works in a busy clinic in a rural hospital in India. Patients are lined up outside her door in a chaotic line all of the way down the hall. Ravdeep is responsible for taking blood pressures and blood sugar measures for each patient. She quickly enters their data into the Simple app on her tablet, and sends them to the doctor if they appear to be hypertensive or diabetic.
This public hospital isn’t fancy, but you might be surprised to know that Ravdeep’s clinic has excellent 4G internet. In fact, much of India is covered with strong, affordable mobile networks. So, why would we choose to make an app like Simple using an offline-first approach?
Offline-first has several upsides, but the primary benefit is that the app is always snappy and responsive, so clinicians aren’t waiting for their app to talk to the cloud during patient interactions. In a busy clinic like Ravdeep’s, this is a crucial feature. In countries like India, Bangladesh, or Ethiopia where a patient encounter is often less than 5 minutes, every second counts and therefore every call to the server matters.
The big secondary benefit is that the app will work in the corners of India with limited internet. In a country as diverse as India, this is crucial.
A typical clinical app is in constant contact with a remote server, so each time patient data is updated it gets saved on the server. Designing an app offline-first is different because that data is saved locally on-device and is later “synced” or saved to a remote server. That means that the app will always be able to function with or without online internet. But, it also means that you need two databases with one on the device itself and one on the server and they must be kept in sync.
The Simple app is being used in over 400 hospitals in India to manage over 190,000 patients with hypertension. We are constantly learning. Here are a few lessons we learned so far:
UUIDs are your friends
Since the records are created on the mobile app and synced to server, we need an identifier that is easy to generate and is unique across all the devices. This makes UUIDs an ideal choice for our primary keys.
Tracking time is really important
In an offline first app, the changes made by a user may not be reflected on the server for a long time, which results in old changes overwriting newer ones. To prevent this, we track when the change was made on the mobile app and the time when the change was seen by the server. With both times we can determine and keep the relevant changes. This is also known as bi-temporal modelling and this blog post provides a nice introduction to the concept.
Limit what data gets synced
When we started, every time a user registered with Simple we would sync all of the data from the server starting from the earliest record. This worked fine when we only had a few facilities, but as the number of patients in the database grew, we started facing two problems:
To improve the user experience we now sync data only for facilities within their district and we sync data for the user’s own facility first. This reduces the initial load and lets the clinician to start recording her patients quickly.
Going offline-first was one of the best decisions we made in our technical architecture. When we watch a nurse in a busy outpatient department, it’s apparent how crucial performance is to her day. We have seen clinicians treating up to 200 patients in a single day — every extra second in a patient interaction at that velocity multiplies quickly.
Committing to offline-first also has opened up more deployment opportunities for Simple. In the next few months we will deploy to several locations where internet access is inconsistent. When our government partners asked if we could handle these environments, we could confidently say “yes!”
Simple is a free, open source, project through Resolve to Save Lives. If you are working on similar software and you have questions about our offline first approach, look at our code and documentation and get in touch with us. We’ll be happy to chat with you and explain in more detail what we’re doing. Thanks!
Offline-first apps are appropriate for many clinical environments was originally published in Simple on Medium, where people are continuing the conversation by highlighting and responding to this story.
]]>One year ago (in October 2018), we piloted the Simple app in five public health facilities in Punjab, India. The original process of teaching healthcare workers to use Simple in pilot hospitals involved in-person trainings from members of our team. We flew from Bangalore to Bathinda, spent a week visiting one facility each day, and taught healthcare workers how to use the app. We’ve come a long way since then.
Simple is now used in over four hundred health facilities in two states. Intensive in-person training is hard to scale, so we had to figure out how to make it easier.
Most other clinical software requires significant investment in training. One goal of Simple is to be able to get a healthcare worker up and running with minimal assistance, within an hour of installing the app. It’s a high bar, so we tried many approaches to explain Simple to our users.
In brief: A short, instructional training video with examples worked best by a long shot, but here are some other ideas we tried.
Our first attempt at onboarding had a basic ‘tour’ with static, abstract screenshots of the functions of Simple.
When our user tester Tanushree Jindal watched users trying the app for the first time, she observed that most users went straight for the “Get started” button, without scrolling through the screenshots that we had put together. Not great.
We created a quick handbook that we could leave behind at in-person trainings. The handbook was pretty helpful, but keeping it updated was a challenge. Besides, they were not easily accessible unless we hand-delivered them.
We then came up with the idea of using ‘coach marks’ to guide the user through the functions of the app. Maybe we could introduce each feature contextually to our users?
We ran extensive usability tests with nurses and realised that there were simply too many coach marks to read. Coach marks were either ignored or got in the way. This didn’t work well either.
During field visits, we identified that a brief, engaging training video with demonstrations could have a lot of impact. So we quickly recorded a friendly video where we enacted the key activities in Simple. This worked really well during user tests and we learned:
We edited the original video down to a crisp 5-minute film that showed a healthcare worker (me!) caring for a patient with hypertension at their clinic. This really hit home, as it was a very relatable setting for the primary users of our app.
We plan to make the recording available in multiple regional languages, so users can watch it in the language of their choice.
Today, the video is part of our short, in-app onboarding process for Simple. A nurse or a doctor can install the app from the Play Store, watch the training video as part of the registration flow, and learn how to use Simple.
The video is always available inside the app, within the built-in HELP section. One key benefit is that the video is nice, but not too fancy. As Simple evolves, we can easily replace it with an updated film that is more relevant.
In addition to users, trainers have given us very positive feedback on the video, as it has proven to be a valuable tool for in-person trainings. Program managers, who go to the point of care to guide users on how the India Hypertension Control Initiative program should be implemented, play the video to demonstrate how Simple fits into the clinic workflows.
Our takeaway from this exercise? Every mode of training had merits, but a contextual demonstration is a powerful tool that seems to hold a user’s attention. We have since recorded a few other videos that demonstrate specific flows in the app, and these have been received with equal enthusiasm.
The goal for the program is for it to eventually be entirely self-sufficient. We believe that by empowering users to teach themselves, we are taking the first important steps in that direction.
Thanks to Daniel Burka for scripting the first draft, and Anand Rama Krishnan for making us look good in the video. Also, thank you to Akshay Verma and Tanushree Jindal for user testing in the field. :)
Training, the Simple way was originally published in Simple on Medium, where people are continuing the conversation by highlighting and responding to this story.
]]>I had gotten comfortable with my usage of pads.
I had always known (at least whole of my 20s) that using pads is ecologically damaging. I am regularly creating non-compostable, non-recyclable waste that is probably burnt or ends up in a landfill. But that is not what triggered me to make this change. It was the rashes, the irritation, the discomfort, and the smell.
It took me close to 10 months to make the switch. I bought my menstrual cup in September 2018. I started using it on and off. At first, my worry was leakage. I got over that in the first month by wearing a panty liner with the cup whenever I was worried. It took a couple of months, but eventually I stopped worrying about that and using liners. Then my worry was using it day to day, in office, in public spaces, specially emptying it or washing it in a public bathroom. Luckily, I am not a heavy bleeder and slowly I got used to using for a stretch of time without checking on it every few hours. And today finally, I am using it outside in an office space without giving it a second thought. It feels like an accomplishment, to be honest. No more unmanaged garbage, no more stealth black polybag or brown paper bag wrapped pad buying from middle aged men and women at pharmacies who think just a glimpse of the pad to the outside world can cause a natural disaster.
A lot of people I know got comfortable with a cup quite easily. One of the reasons I wanted to write to this is to say that it is okay to take your time. If it feels uncomfortable in the beginning, give it a little bit of time, try a different size maybe. Don’t get discouraged and chalk it up to “it is not my cup of tea”. Use it once in a while on your 3rd or 4th day when the flow is decreased. Practice cleaning it in the comfort of your home before you try wearing it outside. Read about other people’s experience or talk to them to see how they went about using it. It is definitely worth the effort, I can say that for sure.
I didn’t do a lot of research into choosing my cup. It was an impulse buy, really. I saw it in a storefront (not in India) and asked the lady in the store a couple of questions before buying it. But post buying it, I read a lot about how to use, how to choose etc before I started using it. I found the information on the Mooncup website(https://www.mooncup.co.uk/) really helpful. You can read that or even ask questions here and I will try to answer them as much as possible.
While I was transitioning from pads to cup, I started using Carmesi (https://mycarmesi.com/) instead of Whisper and that helped me a lot as it is all-natural, completely biodegradable sanitary pads. If you are unsure about the cup, and want to continue using pads — switch to Carmesi.
I have never written a post before so please pardon my writing. Yes, it can happen. I am 28, a software developer, and not a blogger. I just wanted to share this small little accomplishment in the hopes that it might inspire someone else out there to make the switch. Hope it does. :)
]]>When I moved to Bangalore six years ago, I had no interest in India’s garbage management problem. I moved here to build a software company, not to muck about with garbage bins. But every time I would fly between the subcontinent and the Americas, I would find relief on my home continent and exhaustion on my continent of immigration. Living in a giant garbage heap is mentally taxing. Every moment spent outdoors is either an angsty mental rundown of how I might help, complete with acute feelings of powerlessness, or searching for someone to blame (my brain’s lazy personal favourite is a generic and completely unhelpful “the middle class”).
Reprieve comes in the form of resignation as the mind finally comes to accept a dangerous and unsustainable situation as inevitable. Telling yourself “this is just the way it is” at least allows you to enjoy a cup of tea or a single workday without obsessing about the garbage fire you will inevitably inhale on the bicycle ride home from the office.
But that doesn’t work for very long. The workday eventually ends and that bicycle ride inevitably happens. As you hold your breath and ride through the plume of smoke from a smouldering pile of trash on the side of the road, you ask yourself why in the sweet fuck do I even live here? And the answers instantly follow. Your friends are here, your home is here, your job is here. On the smallest scales some part of you belongs in India and on the largest possible scale you recognize that India’s problems of today are Earth’s problems of tomorrow. You can’t ignore them forever.
So you sit down and think about those problems. A lot.
It may seem like a silly question but all sorts of variations have been asked of me over the years. “Why does garbage stink?” “Why can’t we just focus more attention on reuse?” and “Where do the costs of sanitation come from?” come to mind. If you’ve ever thrown something in the trash bin without asking yourself just where is this going to end up? you don’t really know what garbage is. The answer to this question changes with almost every piece of trash, with almost every bin, and definitely with every city.
Garbage is anything society’s majority decides it doesn’t want anymore. This extremely poisonous polystyrene Starbucks lid has served its purpose! I have a litre of burnt coffee in my stomach which means it’s definitely time to poop. Thanks for the assistance and off you go, immortal little lid, to know every worm that crawls by you until the sun of the milky way engulfs the Earth in five billion years and finally incinerates you. While this may be the future the majority plans, our little lid might never make it there. The lowest classes of human society repurpose garbage, sell garbage, and burn garbage to keep warm. The convenience of the healthy, educated elite are a disgrace to our species but, unexpectedly, the response of those at the bottom of the pyramid actually informs the way we should behave. This is because they know what garbage is.
One difficulty with living in a rich country backed by a huge resource economy is that it’s pretty easy to get self-righteous about how people in that country live, even if that way of life is far from perfect. It wasn’t that long ago that the garbage dump outside my hometown still burnt most of its garbage in open fires. (No one in southwest Saskatchewan bothers with that “landfill” euphemism.) It was on one of my many trips back and forth between India and Canada that it struck me how poorly we deal with waste in Canada.
Setting an example matters. If a country with the resources, time, money, and literacy of Canada can’t figure out what to do with our plastic bags or Styrofoam (Thermocol) it’s hard to imagine how any other country could do it. Canada leans on its unbelievable land mass and access to cheap fuel when it chooses to dump garbage in holes. As the world’s population distributes itself more evenly over the coming generations the sweep-it-under-the-rug approach will quickly bring the problems which plague India today to every other country on the planet. The problems of India today are the problems of Earth tomorrow.
Because Level 4 countries like Canada generally don’t appreciate garbage, Canada needs countries like India and China to show it the way. Canada has 3.7 crore (37 million) citizens; India has 134 crore (1.3 billion). Canada’s country-wide garbage solutions need to scale thirty six times over or they are a failure on a global scale.
Thankfully, India’s garbage crisis is a forcing function. Every day the garbage piles up inside and around India’s cities, the weight of it poisons us. Garbage on the street is an opportunity for the average citizen to stop and think about where her plastic milk packets go after they leave her house. Paradoxically, this is India’s opportunity to lead the rest of the globe.
As an individual human being, there are only two steps to understanding our garbage. First, we need to deconstruct it. Second, we need to identify each component.
My friend’s question why does garbage stink? has an answer he could have puzzled out himself and it is arguably the first component of our deconstruction: some garbage is organic. Things we consume which were once alive are still alive on a microscopic level when they come to us. Living things stink. Or they will, eventually. The nice thing about this entire category is that every single item in it falls into the broad category of Organic. Biodegradable. Compost.
In India, biodegradable is often dubbed wet waste but I have issues with that nomenclature. It confuses people. A plastic tray with panipuri juice in it is certainly wet but it is in no way wet waste. If people understand that the green bin only means biodegradable, they’re less likely to throw plastic in there. Prior to our most recent train journey we took note of the waste signage in the Chennai Central railway station. The signage had other issues (which we’ll come to later) but the translations were on-point: biodegradable waste is सड़नशील कचरा in Hindi and… well, Preethi tells me those other words are the right ones in Tamil, whatever they say. The value of the word biodegradable is that it elevates the conversation. Are supposedly biodegradable plastics okay? Are coconut shells biodegradable enough, since they’re more like wood than food? The Wet Waste terminology doesn’t beg these questions.
Our second component comes from the other end of the biodegradability spectrum: plastics. The ubiquitous blue bin has come to mean recyclable material in general but it is plastic that stands apart from all other materials. No matter how many times you reuse that plastic bag, plastic chopstick, or plastic toy it will eventually have an end and that end should neither be the garbage dump nor a trash fire. Plastic needs to be recycled or broken down or we live with it forever.
This is where India gets stuck. There are many different methods of deconstruction and most of them don’t work. Any system which acknowledges the importance of our first two categories (organic and plastic) inherently creates a third: Everything Else.
This often stems from an enthusiasm to solve the garbage crisis. A city like Jaipur sees the problem of unsegregated waste and attacks it head-on… with the wrong weapon.
We recently visited the Jaipur Literature Festival and found ourselves pleasantly surprised by the city’s ubiquitous, segregated public waste system. Jaipur has segregated bins everywhere… but only blue and green bins and only blue and green garbage trucks. Blue/Green, Two-Body segregated bins are a step in the right direction. The citizens of Jaipur see these green and blue bins every day. Whether they respect the bins’ colours or not, a part of their subconscious starts to acknowledge that not all garbage is created equal.
A single look in these bins, however, will show you just how horribly ineffective a Two-Body system is. The green bin is full of dirty styrofoam plates. The blue bin is full of dirty, mixed waste. These broad, mutually exclusive strokes of Green = Wet Waste and Blue = Dry Waste mean that we must choose one of these two categories for Everything Else. A wet plastic dish full of panipuri juice is the least of our troubles. Where should I put a snot-filled tissue? A dirty diaper? Roadkill? Paint chips? Construction rubble? A used sanitary napkin?
Yeah. I don’t know, either. You need three bins.
This question of “other garbage” helped us persuade the building management of our Bangalore apartment complex to let us install a three-bin system: green for organic, blue for recyclables, red for everything else. “Everything else” is labelled reject on our bin but that’s just a fancy way of saying old-fashioned garbage. We actually encouraged our building-mates to put everything in the red bin until they were absolutely certain it belonged in the green or blue bin. A whole bin of clean compost is worthless if a single person poisons it. A whole bin of clean recycling is worthless if a single person dumps chutney all over it.
For months, before we were allowed to install the bins, the building manager repeatedly insisted that “a three-bin system will be too complicated” and “the municipal corporation waste-pickers don’t know what to do with three different bins.”
Over the years before Preethi and I installed these bins, we had heard every brand of argument against segregating garbage: it’s too complicated, the waste-pickers just throw it into one pile anyway, the system is corrupt, the residents / staff / citizens won’t follow the rules, and on and on and on. I’ve fought this battle so many times I’ve even codified my arguments for a Waste Maturity Model into a presentation. Thanks to battling these talking points with our families, friends, and coworkers, we were well-equipped to fend off the argument that the combination of a green and blue bin would be sufficient.
In the end, where does a diaper go? You need three bins.
On our way back from Jaipur to Chennai, we flew through Bangalore and it gave us a chance to see three different cities’ broken waste segregation messages.
Bangalore actually has brilliant state-wide waste segregation guidelines, published in a dozen languages. They answer questions I wouldn’t think to ask (where do my coconut shells go?) and they are actually written into law.
Despite the law, the Bangalore International Airport thinks a 4-bin system is better than a 3-bin system. And, much to our chagrin, none of those bins is for reject garbage. Where do I put my snotty tissue in the Bangalore airport? Is a snotty tissue food, plastic, paper, or cans? None of the above. Hold onto it until you get to Chennai, I guess.
Chennai, on the other hand, has functional segregation at the airport. Hooray! For some unknown reason, they eschewed the standard international green/blue/black colours (or the slightly modified green/blue/red used in Karnataka). But that’s okay. At least I know where to dump my half-eaten dosa, the plastic spoon the chutney came with, and the wrapper with chocolate all over it that melted in my pocket during the flight.
Unfortunately, as soon as you get into the city the only bins you will find are for “non-recyclable waste” — giant green dumpsters akin to those of mid-twentieth-century American cities. If you want to figure out compost or recycling as a citizen of Chennai, you’re on your own.
Moving from one city to another can often be as drastic as moving from one country to another but moving from the airport to the office shouldn’t be. It’s little wonder that citizens are confused, waste-pickers are frustrated, and government officials feel overwhelmed.
It is the perfect time for Indian states to unify on their understanding of the top three categories of waste. Whether we choose to label them wet, dry, and reject or organic, recyclable, and mixed or green, blue, and red or any other three titles, unity from the top will help us pick away at the many difficulties plaguing India’s streets and water bodies.
Over time, these three categories will spawn other categories. Separating plastics from paper from metal from glass from e-waste is a good way to split up the blue bin full of generic recyclables. If you want to split up the city’s plastic, there are at least seven plastic recycling symbols to go by. There are over forty international recycling symbols.
Baby steps, though. Keep the plastic out of the green bin and the chicken grease out of the blue bin. Then maybe we can discuss the road to Zero Waste.
Waste management is a form of project management. The project is massive, distributed, and constantly moving. It is global in scale and influenced by every single human being on Earth. Once it happens, successful waste management in India and China will be the gold standard for the rest of the globe.
These are the biggest projects. If we can do it here, we can do it anywhere.
India has a Three-Body Problem was originally published in Siggu on Medium, where people are continuing the conversation by highlighting and responding to this story.
]]>This is the third post in a series of posts:
Discuss this post on r/haskell.
Sudoku is a number placement puzzle. It consists of a 9x9 grid which is to be filled with digits from 1 to 9 such that each row, each column and each of the nine 3x3 sub-grids contain all the digits. Some of the cells of the grid come pre-filled and the player has to fill the rest.
In the previous post, we improved the performance of the simple Sudoku solver by implementing a new strategy to prune cells. This new strategy found the digits which occurred uniquely, in pairs, or in triplets and fixed the cells to those digits. It led to a speedup of about 200x over our original naive solution. This is our current run1 time for solving all the 49151 17-clue puzzles:
$ cat sudoku17.txt | time stack exec sudoku > /dev/null
258.97 real 257.34 user 1.52 sys
Let’s try to improve this time.2
Instead of trying to guess how to improve the performance of our solution, let’s be methodical about it. We start with profiling the code to find the bottlenecks. Let’s compile and run the code with profiling flags:
$ stack build --profile
$ head -1000 sudoku17.txt | stack exec -- sudoku +RTS -p > /dev/null
This generates a sudoku.prof
file with the profiling output. Here are the top seven Cost Centres3 from the file (cleaned for brevity):
Cost Centre | Src | %time | %alloc |
---|---|---|---|
exclusivePossibilities |
Sudoku.hs:(49,1)-(62,26) | 18.9 | 11.4 |
pruneCellsByFixed.pruneCell |
Sudoku.hs:(75,5)-(76,36) | 17.7 | 30.8 |
exclusivePossibilities.\.\ |
Sudoku.hs:55:38-70 | 11.7 | 20.3 |
fixM.\ |
Sudoku.hs:13:27-65 | 10.7 | 0.0 |
== |
Sudoku.hs:15:56-57 | 5.6 | 0.0 |
pruneGrid' |
Sudoku.hs:(103,1)-(106,64) | 5.0 | 6.7 |
pruneCellsByFixed |
Sudoku.hs:(71,1)-(76,36) | 4.5 | 5.0 |
exclusivePossibilities.\ |
Sudoku.hs:58:36-68 | 3.4 | 2.5 |
Cost Centre points to a function, either named or anonymous. Src gives the line and column numbers of the source code of the function. %time and %alloc are the percentages of time spent and memory allocated in the function, respectively.
We see that exclusivePossibilities
and the nested functions inside it take up almost 34% time of the entire run time. Second biggest bottleneck is the pruneCell
function inside the pruneCellsByFixed
function.
We are going to look at exclusivePossibilities
later. For now, it is easy to guess the possible reason for pruneCell
taking so much time. Here’s the code for reference:
pruneCellsByFixed :: [Cell] -> Maybe [Cell]
pruneCellsByFixed cells = traverse pruneCell cells
where
fixeds = [x | Fixed x <- cells]
pruneCell (Possible xs) = makeCell (xs Data.List.\\ fixeds)
pruneCell x = Just x
pruneCell
uses Data.List.\\
to find the difference of the cell’s possible digits and the fixed digits in the cell’s block. In Haskell, lists are implemented as singly linked lists. So, finding the difference or intersection of two lists is O(n2), that is, quadratic asymptotic complexity. Let’s tackle this bottleneck first.
What is a efficient data structure for finding differences and intersections? Why, a Set of course! A Set stores unique values and provides fast operations for testing membership of its elements. If we use a Set to represent the possible values of cells instead of a List, the program should run faster. Since the possible values are already unique (1
–9
), it should not break anything.
Haskell comes with a bunch of Set implementations:
Data.Set
which is a generic data structure implemented as self-balancing binary search tree.Data.HashSet
which is a generic data structure implemented as hash array mapped trie.Data.IntSet
which is a specialized data structure for integer values, implemented as radix tree.However, a much faster implementation is possible for our particular use-case. We can use a BitSet.
A BitSet uses bits to represent unique members of a Set. We map values to particular bits using some function. If the bit corresponding to a particular value is set to 1 then the value is present in the Set, else it is not. So, we need as many bits in a BitSet as the number of values in our domain, which makes is difficult to use for generic problems. But, for our Sudoku solver, we need to store only the digits 1
–9
in the Set, which make BitSet very suitable for us. Also, the Set operations on BitSet are implemented using bit-level instructions in hardware, making them much faster than those on the other data structure listed above.
In Haskell, we can use the Data.Word
module to represent a BitSet. Specifically, we can use the Data.Word.Word16
type which has sixteen bits because we need only nine bits to represent the nine digits. The bit-level operations on Word16
are provided by the Data.Bits
module.
First, we replace List with Word16
in the Cell
type and add a helper function:
data Cell = Fixed Data.Word.Word16
| Possible Data.Word.Word16
deriving (Show, Eq)
setBits :: Data.Word.Word16 -> [Data.Word.Word16] -> Data.Word.Word16
setBits = Data.List.foldl' (Data.Bits..|.)
Then we replace Int
related operations with bit related ones in the read and show functions:
readGrid :: String -> Maybe Grid
readGrid s
| length s == 81 =
traverse (traverse readCell) . Data.List.Split.chunksOf 9 $ s
| otherwise = Nothing
where
allBitsSet = 1022
readCell '.' = Just $ Possible allBitsSet
readCell c
| Data.Char.isDigit c && c > '0' =
Just . Fixed . Data.Bits.bit . Data.Char.digitToInt $ c
| otherwise = Nothing
showGrid :: Grid -> String
showGrid = unlines . map (unwords . map showCell)
where
showCell (Fixed x) = show . Data.Bits.countTrailingZeros $ x
showCell _ = "."
showGridWithPossibilities :: Grid -> String
showGridWithPossibilities = unlines . map (unwords . map showCell)
where
showCell (Fixed x) = (show . Data.Bits.countTrailingZeros $ x) ++ " "
showCell (Possible xs) =
"[" ++
map (\i -> if Data.Bits.testBit xs i
then Data.Char.intToDigit i
else ' ')
[1..9]
++ "]"
We set the same bits as the digits to indicate the presence of the digits in the possibilities. For example, for digit 1
, we set the bit 1 so that the resulting Word16
is 0000 0000 0000 0010
or 2. This also means, for fixed cells, the value is count of the zeros from right.
The change in the exclusivePossibilities
function is pretty minimal:
-exclusivePossibilities :: [Cell] -> [[Int]]
+exclusivePossibilities :: [Cell] -> [Data.Word.Word16]
exclusivePossibilities row =
row
& zip [1..9]
& filter (isPossible . snd)
& Data.List.foldl'
(\acc ~(i, Possible xs) ->
- Data.List.foldl'
- (\acc' x -> Map.insertWith prepend x [i] acc')
- acc
- xs)
+ Data.List.foldl'
+ (\acc' x -> if Data.Bits.testBit xs x
+ then Map.insertWith prepend x [i] acc'
+ else acc')
+ acc
+ [1..9])
Map.empty
& Map.filter ((< 4) . length)
& Map.foldlWithKey' (\acc x is -> Map.insertWith prepend is [x] acc) Map.empty
& Map.filterWithKey (\is xs -> length is == length xs)
& Map.elems
+ & map (Data.List.foldl' Data.Bits.setBit Data.Bits.zeroBits)
where
prepend ~[y] ys = y:ys
In the nested folding step, instead of folding over the possible values of the cells, now we fold over the digits from 1
to 9
and insert the entry in the map if the bit corresponding to the digit is set in the possibilities. And as the last step, we convert the exclusive possibilities to Word16
by folding them, starting with zero. As example in the REPL should be instructive:
*Main> poss = Data.List.foldl' Data.Bits.setBit Data.Bits.zeroBits
*Main> row = [Possible $ poss [4,6,9], Fixed $ poss [1], Fixed $ poss [5], Possible $ poss [6,9], Fixed $ poss [7], Possible $ poss [2,3,6,8,9], Possible $ poss [6,9], Possible $ poss [2,3,6,8,9], Possible $ poss [2,3,6,8,9]]
*Main> putStr $ showGridWithPossibilities [row]
[ 4 6 9] 1 5 [ 6 9] 7 [ 23 6 89] [ 6 9] [ 23 6 89] [ 23 6 89]
*Main> exclusivePossibilities row
[16,268]
*Main> [poss [4], poss [8,3,2]]
[16,268]
This is the same example row as the last time. And it returns same results, excepts as a list of Word16
now.
Now, we change makeCell
to use bit operations instead of list ones:
makeCell :: Data.Word.Word16 -> Maybe Cell
makeCell ys
| ys == Data.Bits.zeroBits = Nothing
| Data.Bits.popCount ys == 1 = Just $ Fixed ys
| otherwise = Just $ Possible ys
And we change cell pruning functions too:
pruneCellsByFixed :: [Cell] -> Maybe [Cell]
pruneCellsByFixed cells = traverse pruneCell cells
where
- fixeds = [x | Fixed x <- cells]
+ fixeds = setBits Data.Bits.zeroBits [x | Fixed x <- cells]
- pruneCell (Possible xs) = makeCell (xs Data.List.\\ fixeds)
+ pruneCell (Possible xs) =
+ makeCell (xs Data.Bits..&. Data.Bits.complement fixeds)
pruneCell x = Just x
pruneCellsByExclusives :: [Cell] -> Maybe [Cell]
pruneCellsByExclusives cells = case exclusives of
[] -> Just cells
_ -> traverse pruneCell cells
where
exclusives = exclusivePossibilities cells
- allExclusives = concat exclusives
+ allExclusives = setBits Data.Bits.zeroBits exclusives
pruneCell cell@(Fixed _) = Just cell
pruneCell cell@(Possible xs)
| intersection `elem` exclusives = makeCell intersection
| otherwise = Just cell
where
- intersection = xs `Data.List.intersect` allExclusives
+ intersection = xs Data.Bits..&. allExclusives
Notice how the list difference and intersection functions are replaced by Data.Bits
functions. Specifically, list difference is replace by bitwise-and of the bitwise-complement, and list intersection is replaced by bitwise-and.
We make a one-line change in the isGridInvalid
function to find empty possible cells using bit ops:
isGridInvalid :: Grid -> Bool
isGridInvalid grid =
any isInvalidRow grid
|| any isInvalidRow (Data.List.transpose grid)
|| any isInvalidRow (subGridsToRows grid)
where
isInvalidRow row =
let fixeds = [x | Fixed x <- row]
- emptyPossibles = [x | Possible x <- row, null x]
+ emptyPossibles = [() | Possible x <- row, x == Data.Bits.zeroBits]
in hasDups fixeds || not (null emptyPossibles)
hasDups l = hasDups' l []
hasDups' [] _ = False
hasDups' (y:ys) xs
| y `elem` xs = True
| otherwise = hasDups' ys (y:xs)
And finally, we change the nextGrids
functions to use bit operations:
nextGrids :: Grid -> (Grid, Grid)
nextGrids grid =
let (i, first@(Fixed _), rest) =
fixCell
. Data.List.minimumBy (compare `Data.Function.on` (possibilityCount . snd))
. filter (isPossible . snd)
. zip [0..]
. concat
$ grid
in (replace2D i first grid, replace2D i rest grid)
where
possibilityCount (Possible xs) = Data.Bits.popCount xs
possibilityCount (Fixed _) = 1
fixCell ~(i, Possible xs) =
let x = Data.Bits.countTrailingZeros xs
in case makeCell (Data.Bits.clearBit xs x) of
Nothing -> error "Impossible case"
Just cell -> (i, Fixed (Data.Bits.bit x), cell)
replace2D :: Int -> a -> [[a]] -> [[a]]
replace2D i v =
let (x, y) = (i `quot` 9, i `mod` 9) in replace x (replace y (const v))
replace p f xs = [if i == p then f x else x | (x, i) <- zip xs [0..]]
possibilityCount
now uses Data.Bits.popCount
to count the number of bits set to 1. fixCell
now chooses the first set bit from right as the digit to fix. Rest of the code stays the same. Let’s build and run it:
$ stack build
$ cat sudoku17.txt | time stack exec sudoku > /dev/null
69.44 real 69.12 user 0.37 sys
Wow! That is almost 3.7x faster than the previous solution. It’s a massive win! But let’s not be content yet. To the profiler again!4
Running the profiler again gives us these top six culprits:
Cost Centre | Src | %time | %alloc |
---|---|---|---|
exclusivePossibilities |
Sudoku.hs:(57,1)-(74,26) | 25.2 | 16.6 |
exclusivePossibilities.\.\ |
Sudoku.hs:64:23-96 | 19.0 | 32.8 |
fixM.\ |
Sudoku.hs:15:27-65 | 12.5 | 0.1 |
pruneCellsByFixed |
Sudoku.hs:(83,1)-(88,36) | 5.9 | 7.1 |
pruneGrid' |
Sudoku.hs:(115,1)-(118,64) | 5.0 | 8.6 |
Hurray! pruneCellsByFixed.pruneCell
has disappeared from the list of top bottlenecks. Though exclusivePossibilities
still remains here as expected.
exclusivePossibilities
is a big function. The profiler does not really tell us which parts of it are the slow ones. That’s because by default, the profiler only considers functions as Cost Centres. We need to give it hints for it to be able to find bottlenecks inside functions. For that, we need to insert Cost Centre annotations in the code:
exclusivePossibilities :: [Cell] -> [Data.Word.Word16]
exclusivePossibilities row =
row
& ({-# SCC "EP.zip" #-} zip [1..9])
& ({-# SCC "EP.filter" #-} filter (isPossible . snd))
& ({-# SCC "EP.foldl" #-} Data.List.foldl'
(\acc ~(i, Possible xs) ->
Data.List.foldl'
(\acc' n -> if Data.Bits.testBit xs n
then Map.insertWith prepend n [i] acc'
else acc')
acc
[1..9])
Map.empty)
& ({-# SCC "EP.Map.filter1" #-} Map.filter ((< 4) . length))
& ({-# SCC "EP.Map.foldl" #-}
Map.foldlWithKey'
(\acc x is -> Map.insertWith prepend is [x] acc)
Map.empty)
& ({-# SCC "EP.Map.filter2" #-}
Map.filterWithKey (\is xs -> length is == length xs))
& ({-# SCC "EP.Map.elems" #-} Map.elems)
& ({-# SCC "EP.map" #-}
map (Data.List.foldl' Data.Bits.setBit Data.Bits.zeroBits))
where
prepend ~[y] ys = y:ys
Here, {-# SCC "EP.zip" #-}
is a Cost Centre annotation. "EP.zip"
is the name we choose to give to this Cost Centre.
After profiling the code again, we get a different list of bottlenecks:
Cost Centre | Src | %time | %alloc |
---|---|---|---|
exclusivePossibilities.\.\ |
Sudoku.hs:(64,23)-(66,31) | 19.5 | 31.4 |
fixM.\ |
Sudoku.hs:15:27-65 | 13.1 | 0.1 |
pruneCellsByFixed |
Sudoku.hs:(85,1)-(90,36) | 5.4 | 6.8 |
pruneGrid' |
Sudoku.hs:(117,1)-(120,64) | 4.8 | 8.3 |
EP.zip |
Sudoku.hs:59:27-36 | 4.3 | 10.7 |
EP.Map.filter1 |
Sudoku.hs:70:35-61 | 4.2 | 0.5 |
chunksOf |
Data/List/Split/Internals.hs:(514,1)-(517,49) | 4.1 | 7.4 |
exclusivePossibilities.\ |
Sudoku.hs:71:64-96 | 4.0 | 3.4 |
EP.filter |
Sudoku.hs:60:30-54 | 2.9 | 3.4 |
EP.foldl |
Sudoku.hs:(61,29)-(69,15) | 2.8 | 1.8 |
exclusivePossibilities |
Sudoku.hs:(57,1)-(76,26) | 2.7 | 1.9 |
chunksOf.splitter |
Data/List/Split/Internals.hs:(516,3)-(517,49) | 2.5 | 2.7 |
So almost one-fifth of the time is actually going in this nested one-line anonymous function inside exclusivePossibilities
:
But we are going to ignore it for now.
If we look closely, we also find that around 17% of the run time now goes into list traversal and manipulation. This is in the functions pruneCellsByFixed
, pruneGrid'
, chunksOf
and chunksOf.splitter
, where the first two are majorly list traversal and transposition, and the last two are list splitting. Maybe it is time to get rid of lists altogether?
Vector is a Haskell library for working with arrays. It implements very performant operations for integer-indexed array data. Unlike the lists in Haskell which are implemented as singly linked lists, vectors are stored in a contiguous set of memory locations. This makes random access to the elements a constant time operation. The memory overhead per additional item in vectors is also much smaller. Lists allocate memory for each item in the heap and have pointers to the memory locations in nodes, leading to a lot of wasted memory in holding pointers. On the other hand, operations on lists are lazy, whereas, operations on vectors are strict, and this may need to useless computation depending on the use-case5.
In our current code, we represent the grid as a list of lists of cells. All the pruning operations require us to traverse the grid list or the row lists. We also need to transform the grid back-and-forth for being able to use the same pruning operations for rows, columns and sub-grids. The pruning of cells and the choosing of pivot cells also requires us to replace cells in the grid with new ones, leading to a lot of list traversals.
To prevent all this linear-time list traversals, we can replace the nested list of lists with a single vector. Then all we need to do it to go over the right parts of this vector, looking up and replacing cells as needed. Since both lookups and updates on vectors are constant time, this should lead to a speedup.
Let’s start by changing the grid to a vector of cells.:
data Cell = Fixed Data.Word.Word16
| Possible Data.Word.Word16
deriving (Show, Eq)
type Grid = Data.Vector.Vector Cell
Since we plan to traverse different parts of the same vector, let’s define these different parts first:
type CellIxs = [Int]
fromXY :: (Int, Int) -> Int
fromXY (x, y) = x * 9 + y
allRowIxs, allColIxs, allSubGridIxs :: [CellIxs]
allRowIxs = [getRow i | i <- [0..8]]
where getRow n = [ fromXY (n, i) | i <- [0..8] ]
allColIxs = [getCol i | i <- [0..8]]
where getCol n = [ fromXY (i, n) | i <- [0..8] ]
allSubGridIxs = [getSubGrid i | i <- [0..8]]
where getSubGrid n = let (r, c) = (n `quot` 3, n `mod` 3)
in [ fromXY (3 * r + i, 3 * c + j) | i <- [0..2], j <- [0..2] ]
We define a type for cell indices as a list of integers. Then we create three lists of cell indices: all row indices, all column indices, and all sub-grid indices. Let’s check these out in the REPL:
*Main> Control.Monad.mapM_ print allRowIxs
[0,1,2,3,4,5,6,7,8]
[9,10,11,12,13,14,15,16,17]
[18,19,20,21,22,23,24,25,26]
[27,28,29,30,31,32,33,34,35]
[36,37,38,39,40,41,42,43,44]
[45,46,47,48,49,50,51,52,53]
[54,55,56,57,58,59,60,61,62]
[63,64,65,66,67,68,69,70,71]
[72,73,74,75,76,77,78,79,80]
*Main> Control.Monad.mapM_ print allColIxs
[0,9,18,27,36,45,54,63,72]
[1,10,19,28,37,46,55,64,73]
[2,11,20,29,38,47,56,65,74]
[3,12,21,30,39,48,57,66,75]
[4,13,22,31,40,49,58,67,76]
[5,14,23,32,41,50,59,68,77]
[6,15,24,33,42,51,60,69,78]
[7,16,25,34,43,52,61,70,79]
[8,17,26,35,44,53,62,71,80]
*Main> Control.Monad.mapM_ print allSubGridIxs
[0,1,2,9,10,11,18,19,20]
[3,4,5,12,13,14,21,22,23]
[6,7,8,15,16,17,24,25,26]
[27,28,29,36,37,38,45,46,47]
[30,31,32,39,40,41,48,49,50]
[33,34,35,42,43,44,51,52,53]
[54,55,56,63,64,65,72,73,74]
[57,58,59,66,67,68,75,76,77]
[60,61,62,69,70,71,78,79,80]
We can verify manually that these indices are correct.
Read and show functions are easy to change for vector:
readGrid :: String -> Maybe Grid
readGrid s
- | length s == 81 = traverse (traverse readCell) . Data.List.Split.chunksOf 9 $ s
+ | length s == 81 = Data.Vector.fromList <$> traverse readCell s
| otherwise = Nothing
where
allBitsSet = 1022
readCell '.' = Just $ Possible allBitsSet
readCell c
| Data.Char.isDigit c && c > '0' =
Just . Fixed . Data.Bits.bit . Data.Char.digitToInt $ c
| otherwise = Nothing
showGrid :: Grid -> String
-showGrid = unlines . map (unwords . map showCell)
+showGrid grid =
+ unlines . map (unwords . map (showCell . (grid !))) $ allRowIxs
where
showCell (Fixed x) = show . Data.Bits.countTrailingZeros $ x
showCell _ = "."
showGridWithPossibilities :: Grid -> String
-showGridWithPossibilities = unlines . map (unwords . map showCell)
+showGridWithPossibilities grid =
+ unlines . map (unwords . map (showCell . (grid !))) $ allRowIxs
where
showCell (Fixed x) = (show . Data.Bits.countTrailingZeros $ x) ++ " "
showCell (Possible xs) =
"[" ++
map (\i -> if Data.Bits.testBit xs i
then Data.Char.intToDigit i
else ' ')
[1..9]
++ "]"
readGrid
simply changes to work on a single vector of cells instead of a list of lists. Show functions have a pretty minor change to do lookups from a vector using the row indices and the (!)
function. The (!)
function is the vector indexing function which is similar to the (!!)
function, except it executes in constant time.
The pruning related functions are rewritten for working with vectors:
replaceCell :: Int -> Cell -> Grid -> Grid
replaceCell i c g = g Data.Vector.// [(i, c)]
pruneCellsByFixed :: Grid -> CellIxs -> Maybe Grid
pruneCellsByFixed grid cellIxs =
Control.Monad.foldM pruneCell grid . map (\i -> (i, grid ! i)) $ cellIxs
where
fixeds = setBits Data.Bits.zeroBits [x | Fixed x <- map (grid !) cellIxs]
pruneCell g (_, Fixed _) = Just g
pruneCell g (i, Possible xs)
| xs' == xs = Just g
| otherwise = flip (replaceCell i) g <$> makeCell xs'
where
xs' = xs Data.Bits..&. Data.Bits.complement fixeds
pruneCellsByExclusives :: Grid -> CellIxs -> Maybe Grid
pruneCellsByExclusives grid cellIxs = case exclusives of
[] -> Just grid
_ -> Control.Monad.foldM pruneCell grid . zip cellIxs $ cells
where
cells = map (grid !) cellIxs
exclusives = exclusivePossibilities cells
allExclusives = setBits Data.Bits.zeroBits exclusives
pruneCell g (_, Fixed _) = Just g
pruneCell g (i, Possible xs)
| intersection == xs = Just g
| intersection `elem` exclusives =
flip (replaceCell i) g <$> makeCell intersection
| otherwise = Just g
where
intersection = xs Data.Bits..&. allExclusives
pruneCells :: Grid -> CellIxs -> Maybe Grid
pruneCells grid cellIxs =
fixM (flip pruneCellsByFixed cellIxs) grid
>>= fixM (flip pruneCellsByExclusives cellIxs)
All the three functions now take the grid and the cell indices instead of a list of cells, and use the cell indices to lookup the cells from the grid. Also, instead of using the traverse
function as earlier, now we use the Control.Monad.foldM
function to fold over the cell-index-and-cell tuples in the context of the Maybe
monad, making changes to the grid directly.
We use the replaceCell
function to replace cells at an index in the grid. It is a simple wrapper over the vector update function Data.Vector.//
. Rest of the code is same in essence, except a few changes to accommodate the changed function parameters.
pruneGrid'
function does not need to do transpositions and back-transpositions anymore as now we use the cell indices to go over the right parts of the grid vector directly:
pruneGrid' :: Grid -> Maybe Grid
pruneGrid' grid =
Control.Monad.foldM pruneCells grid allRowIxs
>>= flip (Control.Monad.foldM pruneCells) allColIxs
>>= flip (Control.Monad.foldM pruneCells) allSubGridIxs
Notice that the traverse
function here is also replaced by the Control.Monad.foldM
function.
Similarly, the grid predicate functions change a little to go over a vector instead of a list of lists:
isGridFilled :: Grid -> Bool
-isGridFilled grid = null [ () | Possible _ <- concat grid ]
+isGridFilled = not . Data.Vector.any isPossible
isGridInvalid :: Grid -> Bool
isGridInvalid grid =
- any isInvalidRow grid
- || any isInvalidRow (Data.List.transpose grid)
- || any isInvalidRow (subGridsToRows grid)
+ any isInvalidRow (map (map (grid !)) allRowIxs)
+ || any isInvalidRow (map (map (grid !)) allColIxs)
+ || any isInvalidRow (map (map (grid !)) allSubGridIxs)
And finally, we change the nextGrids
function to replace the list related operations with the vector related ones:
nextGrids :: Grid -> (Grid, Grid)
nextGrids grid =
let (i, first@(Fixed _), rest) =
fixCell
- . Data.List.minimumBy
+ . Data.Vector.minimumBy
(compare `Data.Function.on` (possibilityCount . snd))
- . filter (isPossible . snd)
- . zip [0..]
- . concat
+ . Data.Vector.imapMaybe
+ (\j cell -> if isPossible cell then Just (j, cell) else Nothing)
$ grid
- in (replace2D i first grid, replace2D i rest grid)
+ in (replaceCell i first grid, replaceCell i rest grid)
We also switch the replace2D
function which went over the entire list of lists of cells to replace a cell, with the vector-based replaceCell
function.
All the required changes are done. Let’s do a run:
$ stack build
$ cat sudoku17.txt | time stack exec sudoku > /dev/null
88.53 real 88.16 user 0.41 sys
Oops! Instead of getting a speedup, our vector-based code is actually 1.3x slower than the list-based code. How did this happen? Time to bust out the profiler again!
(==)
Profiling the current code gives us the following hotspots:
Cost Centre | Src | %time | %alloc |
---|---|---|---|
>>= |
Data/Vector/Fusion/Util.hs:36:3-18 | 52.2 | 51.0 |
basicUnsafeIndexM |
Data/Vector.hs:278:3-62 | 22.2 | 20.4 |
exclusivePossibilities |
Sudoku.hs:(75,1)-(93,26) | 6.8 | 8.3 |
exclusivePossibilities.\.\ |
Sudoku.hs:83:23-96 | 3.8 | 8.8 |
pruneCellsByFixed.fixeds |
Sudoku.hs:105:5-77 | 2.0 | 1.7 |
We see a sudden appearance of (>>=)
from the Data.Vector.Fusion.Util
module at the top of the list, taking more than half of the run time. For more clues, we dive into the detailed profiler report and find this bit:
Cost Centre | Src | %time | %alloc |
---|---|---|---|
pruneGrid |
Sudoku.hs:143:1-27 | 0.0 | 0.0 |
fixM |
Sudoku.hs:16:1-65 | 0.1 | 0.0 |
fixM.\ |
Sudoku.hs:16:27-65 | 0.2 | 0.1 |
== |
Data/Vector.hs:287:3-50 | 1.0 | 1.4 |
>>= |
Data/Vector/Fusion/Util.hs:36:3-18 | 51.9 | 50.7 |
basicUnsafeIndexM |
Data/Vector.hs:278:3-62 | 19.3 | 20.3 |
Here, the indentation indicated nesting of operations. We see that both the (>>=)
and basicUnsafeIndexM
functions — which together take around three-quarter of the run time — are being called from the (==)
function in the fixM
function6. It seems like we are checking for equality too many times. Here’s the usage of the fixM
for reference:
pruneCells :: Grid -> CellIxs -> Maybe Grid
pruneCells grid cellIxs =
fixM (flip pruneCellsByFixed cellIxs) grid
>>= fixM (flip pruneCellsByExclusives cellIxs)
pruneGrid :: Grid -> Maybe Grid
pruneGrid = fixM pruneGrid'
In pruneGrid
, we run pruneGrid'
till the resultant grid settles, that is, the grid computed in a particular iteration is equal to the grid in the previous iteration. Interestingly, we do the same thing in pruneCells
too. We equate the whole grid to check for settling of each block of cells. This is the reason of the slowdown.
Why did we add fixM
in the pruneCells
function at all? Quoting from the previous post,
We need to run
pruneCellsByFixed
andpruneCellsByExclusives
repeatedly usingfixM
because an unsettled row can lead to wrong solutions.Imagine a row which just got a
9
fixed because ofpruneCellsByFixed
. If we don’t run the function again, the row may be left with one non-fixed cell with a9
. When we run this row throughpruneCellsByExclusives
, it’ll consider the9
in the non-fixed cell as a Single and fix it. This will lead to two9
s in the same row, causing the solution to fail.
So the reason we added fixM
is that, we run the two pruning strategies one-after-another. That way, they see the cells in the same block in different states. If we were to merge the two pruning functions into a single one such that they work in lockstep, we would not need to run fixM
at all!
With this idea, we rewrite pruneCells
as a single function:
pruneCells :: Grid -> CellIxs -> Maybe Grid
pruneCells grid cellIxs = Control.Monad.foldM pruneCell grid cellIxs
where
cells = map (grid !) cellIxs
exclusives = exclusivePossibilities cells
allExclusives = setBits Data.Bits.zeroBits exclusives
fixeds = setBits Data.Bits.zeroBits [x | Fixed x <- cells]
pruneCell g i =
pruneCellByFixed g (i, g ! i) >>= \g' -> pruneCellByExclusives g' (i, g' ! i)
pruneCellByFixed g (_, Fixed _) = Just g
pruneCellByFixed g (i, Possible xs)
| xs' == xs = Just g
| otherwise = flip (replaceCell i) g <$> makeCell xs'
where
xs' = xs Data.Bits..&. Data.Bits.complement fixeds
pruneCellByExclusives g (_, Fixed _) = Just g
pruneCellByExclusives g (i, Possible xs)
| null exclusives = Just g
| intersection == xs = Just g
| intersection `elem` exclusives =
flip (replaceCell i) g <$> makeCell intersection
| otherwise = Just g
where
intersection = xs Data.Bits..&. allExclusives
We have merged the two pruning functions almost blindly. The important part here is the nested pruneCell
function which uses monadic bind (>>=)
to ensure that cells fixed in the first step are seen by the next step. Merging the two functions ensures that both strategies will see same Exclusives and Fixeds, thereby running in lockstep.
Let’s try it out:
$ stack build
$ cat sudoku17.txt | time stack exec sudoku > /dev/null
57.67 real 57.12 user 0.46 sys
Ah, now it’s faster than the list-based implementation by 1.2x7. Let’s see what the profiler says:
Cost Centre | Src | %time | %alloc |
---|---|---|---|
exclusivePossibilities.\.\ |
Sudoku.hs:82:23-96 | 15.7 | 33.3 |
pruneCells |
Sudoku.hs:(101,1)-(126,53) | 9.6 | 6.8 |
pruneCells.pruneCell |
Sudoku.hs:(108,5)-(109,83) | 9.5 | 2.1 |
basicUnsafeIndexM |
Data/Vector.hs:278:3-62 | 9.4 | 0.5 |
pruneCells.pruneCell.\ |
Sudoku.hs:109:48-83 | 7.6 | 2.1 |
pruneCells.cells |
Sudoku.hs:103:5-40 | 7.1 | 10.9 |
exclusivePossibilities.\ |
Sudoku.hs:87:64-96 | 3.5 | 3.8 |
EP.Map.filter1 |
Sudoku.hs:86:35-61 | 3.0 | 0.6 |
>>= |
Data/Vector/Fusion/Util.hs:36:3-18 | 2.8 | 2.0 |
replaceCell |
Sudoku.hs:59:1-45 | 2.5 | 1.1 |
EP.filter |
Sudoku.hs:78:30-54 | 2.4 | 3.3 |
primitive |
Control/Monad/Primitive.hs:195:3-16 | 2.3 | 6.5 |
The double nested anonymous function mentioned before is still the biggest culprit but fixM
has disappeared from the list. Let’s tackle exclusivePossibilities
now.
Here’s exclusivePossibilities
again for reference:
exclusivePossibilities :: [Cell] -> [Data.Word.Word16]
exclusivePossibilities row =
row
& zip [1..9]
& filter (isPossible . snd)
& Data.List.foldl'
(\acc ~(i, Possible xs) ->
Data.List.foldl'
(\acc' n -> if Data.Bits.testBit xs n
then Map.insertWith prepend n [i] acc'
else acc')
acc
[1..9])
Map.empty
& Map.filter ((< 4) . length)
& Map.foldlWithKey'(\acc x is -> Map.insertWith prepend is [x] acc) Map.empty
& Map.filterWithKey (\is xs -> length is == length xs)
& Map.elems
& map (Data.List.foldl' Data.Bits.setBit Data.Bits.zeroBits)
where
prepend ~[y] ys = y:ys
Let’s zoom into lines 6–14. Here, we do a fold with a nested fold over the non-fixed cells of the given block to accumulate the mapping from the digits to the indices of the cells they occur in. We use a Data.Map.Strict
map as the accumulator. If a digit is not present in the map as a key then we add a singleton list containing the corresponding cell index as the value. If the digit is already present in the map then we prepend the cell index to the list of indices for the digit. So we end up “mutating” the map repeatedly.
Of course, it’s not actual mutation because the map data structure we are using is immutable. Each change to the map instance creates a new copy with the addition, which we thread through the fold operation, and we get the final copy at the end. This may be the reason of the slowness in this section of the code.
What if, instead of using an immutable data structure for this, we used a mutable one? But how can we do that when we know that Haskell is a pure language? Purity means that all code must be referentially transparent, and mutability certainly isn’t. It turns out, there is an escape hatch to mutability in Haskell. Quoting the relevant section from the book Real World Haskell:
Haskell provides a special monad, named
ST
, which lets us work safely with mutable state. Compared to theState
monad, it has some powerful added capabilities.
- We can thaw an immutable array to give a mutable array; modify the mutable array in place; and freeze a new immutable array when we are done.
- We have the ability to use mutable references. This lets us implement data structures that we can modify after construction, as in an imperative language. This ability is vital for some imperative data structures and algorithms, for which similarly efficient purely functional alternatives have not yet been discovered.
So if we use a mutable map in the ST
monad, we may be able to get rid of this bottleneck. But, we can actually do better! Since the keys of our map are digits 1
–9
, we can use a mutable vector to store the indices. In fact, we can go one step even further and store the indices as a BitSet as Word16
because they also range from 1 to 9, and are unique for a block. This lets us use an unboxed mutable vector. What is unboxing you ask? Quoting from the GHC docs:
Most types in GHC are boxed, which means that values of that type are represented by a pointer to a heap object. The representation of a Haskell
Int
, for example, is a two-word heap object. An unboxed type, however, is represented by the value itself, no pointers or heap allocation are involved.
When combined with vector, unboxing of values means the whole vector is stored as single byte array, avoiding pointer redirections completely. This is more memory efficient and allows better usage of caches8. Let’s rewrite exclusivePossibilities
using ST
and unboxed mutable vectors.
First we write the core of this operation, the function cellIndicesList
which take a list of cells and returns the digit to cell indices mapping. The mapping is returned as a list. The zeroth value in this list is the indices of the cells which have 1
as a possible digit, and so on. The indices themselves are packed as BitSets. If the bit 1 is set then the first cell has a particular digit. Let’s say it returns [0,688,54,134,0,654,652,526,670]
. In 10-bit binary it is:
[0000000000, 1010110000, 0000110110, 0010000110, 0000000000, 1010001110, 1010001100, 1000001110, 1010011110]
We can arrange it in a table for further clarity:
Digits | Cell 9 | Cell 8 | Cell 7 | Cell 6 | Cell 5 | Cell 4 | Cell 3 | Cell 2 | Cell 1 |
---|---|---|---|---|---|---|---|---|---|
1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
2 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 |
3 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 1 | 1 |
4 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 1 |
5 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
6 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 1 |
7 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 |
8 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 |
9 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 1 | 1 |
If the value of the intersection of a particular digit and a particular cell index in the table is set to 1, then the digit is a possibility in the cell, else it is not. Here’s the code:
cellIndicesList :: [Cell] -> [Data.Word.Word16]
cellIndicesList cells =
Data.Vector.Unboxed.toList $ Control.Monad.ST.runST $ do
vec <- Data.Vector.Unboxed.Mutable.replicate 9 Data.Bits.zeroBits
ref <- Data.STRef.newSTRef (1 :: Int)
Control.Monad.forM_ cells $ \cell -> do
i <- Data.STRef.readSTRef ref
case cell of
Fixed _ -> return ()
Possible xs -> Control.Monad.forM_ [0..8] $ \d ->
Control.Monad.when (Data.Bits.testBit xs (d+1)) $
Data.Vector.Unboxed.Mutable.unsafeModify vec (`Data.Bits.setBit` i) d
Data.STRef.writeSTRef ref (i+1)
Data.Vector.Unboxed.unsafeFreeze vec
The whole mutable code runs inside the runST
function. runST
take an operation in ST
monad and executes it, making sure that the mutable references created inside it cannot escape the scope of runST
. This is done using a type-system trickery called Rank-2 types.
Inside the ST
operation, we start with creating a mutable vector of Word16
s of size 9 with all its values initially set to zero. We also initialize a mutable reference to keep track of the cell index we are on. Then we run two nested for loops, going over each cell and each digit 1
–9
, setting the right bit of the right index of the mutable vector. During this, we mutate the vector directly using the Data.Vector.Unboxed.Mutable.unsafeModify
function. At the end of the ST
operation, we freeze the mutable vector to return an immutable version of it. Outside runST
, we convert the immutable vector to a list. Notice how this code is quite similar to how we’d write it in imperative programming languages like C or Java9.
It is easy to use this function now to rewrite exclusivePossibilities
:
exclusivePossibilities :: [Cell] -> [Data.Word.Word16]
exclusivePossibilities row =
row
- & zip [1..9]
- & filter (isPossible . snd)
- & Data.List.foldl'
- (\acc ~(i, Possible xs) ->
- Data.List.foldl'
- (\acc' n -> if Data.Bits.testBit xs n
- then Map.insertWith prepend n [i] acc'
- else acc')
- acc
- [1..9])
- Map.empty
+ & cellIndicesList
+ & zip [1..9]
- & Map.filter ((< 4) . length)
- & Map.foldlWithKey' (\acc x is -> Map.insertWith prepend is [x] acc) Map.empty
- & Map.filterWithKey (\is xs -> length is == length xs)
+ & filter (\(_, is) -> let p = Data.Bits.popCount is in p > 0 && p < 4)
+ & Data.List.foldl' (\acc (x, is) -> Map.insertWith prepend is [x] acc) Map.empty
+ & Map.filterWithKey (\is xs -> Data.Bits.popCount is == length xs)
& Map.elems
& map (Data.List.foldl' Data.Bits.setBit Data.Bits.zeroBits)
where
prepend ~[y] ys = y:ys
We replace the nested two-fold operation with cellIndicesList
. Then we replace some map related function with the corresponding list ones because cellIndicesList
returns a list. We also replace the length
function call on cell indices with Data.Bits.popCount
function call as the indices are represented as Word16
now.
That is it. Let’s build and run it now:
$ stack build
$ cat sudoku17.txt | time stack exec sudoku > /dev/null
35.04 real 34.84 user 0.24 sys
That’s a 1.6x speedup over the map-and-fold based version. Let’s check what the profiler has to say:
Cost Centre | Src | %time | %alloc |
---|---|---|---|
cellIndicesList.\.\ |
Sudoku.hs:(88,11)-(89,81) | 10.7 | 6.0 |
primitive |
Control/Monad/Primitive.hs:195:3-16 | 7.9 | 6.9 |
pruneCells |
Sudoku.hs:(113,1)-(138,53) | 7.5 | 6.4 |
cellIndicesList |
Sudoku.hs:(79,1)-(91,40) | 7.4 | 10.1 |
basicUnsafeIndexM |
Data/Vector.hs:278:3-62 | 7.3 | 0.5 |
pruneCells.pruneCell |
Sudoku.hs:(120,5)-(121,83) | 6.8 | 2.0 |
exclusivePossibilities |
Sudoku.hs:(94,1)-(104,26) | 6.5 | 9.7 |
pruneCells.pruneCell.\ |
Sudoku.hs:121:48-83 | 6.1 | 2.0 |
cellIndicesList.\ |
Sudoku.hs:(83,42)-(90,37) | 5.5 | 3.5 |
pruneCells.cells |
Sudoku.hs:115:5-40 | 5.0 | 10.4 |
The run time is spread quite evenly over all the functions now and there are no hotspots anymore. We stop optimizating at this point10. Let’s see how far we have come up.
Below is a table showing the speedups we got with each new implementation:
Implementation | Run Time (s) | Incremental Speedup | Cumulative Speedup |
---|---|---|---|
Simple | 47450 | 1x | 1x |
Exclusive Pruning | 258.97 | 183.23x | 183x |
BitSet | 69.44 | 3.73x | 683x |
Vector | 57.67 | 1.20x | 823x |
Mutable Vector | 35.04 | 1.65x | 1354x |
The first improvement over the simple solution got us the most major speedup of 183x. After that, we followed the profiler, fixing bottlenecks by using the right data structures. We got quite significant speedup over the naive list-based solution, leading to drop in the run time from 259 seconds to 35 seconds. In total, we have done more than a thousand times improvement in the run time since the first solution!
In this post, we improved upon our list-based Sudoku solution from the last time. We profiled the code at each step, found the bottlenecks and fixed them by choosing the right data structure for the case. We ended up using BitSets and Vectors — both immutable and mutable varieties — for the different parts of the code. Finally, we sped up our program by 7.4 times. Can we go even faster? How about using all those other CPU cores which have been lying idle? Come back for the next post in this series where we’ll explore the parallel programming facilities in Haskell. The code till now is available here. Discuss this post on r/haskell or leave a comment.
All the runs were done on my MacBook Pro from 2014 with 2.2 GHz Intel Core i7 CPU and 16 GB memory.↩︎
A lot of the code in this post references the code from the previous posts, including showing diffs. So, please read the previous posts if you have not already done so.↩︎
Notice the British English spelling of the word “Centre”. GHC was originally developed in University of Glasgow in Scotland.↩︎
The code for the BitSet based implementation can be found here.↩︎
This article on School of Haskell goes into details about performance of vectors vs. lists. There are also these benchmarks for sequence data structures in Haskell: lists, vectors, seqs, etc.↩︎
We see Haskell’s laziness at work here. In the code for the fixM
function, the (==)
function is nested inside the (>>=)
function, but because of laziness, they are actually evaluated in the reverse order. The evaluation of parameters for the (==)
function causes the (>>=)
function to be evaluated.↩︎
The code for the vector based implementation can be found here.↩︎
Unboxed vectors have some restrictions on the kind of values that can be put into them but Word16
already follows those restrictions so we are good.↩︎
Haskell can be a pretty good imperative programming language using the ST
monad. This article shows how to implement some algorithms which require mutable data structures in Haskell.↩︎
The code for the mutable vector based implementation can be found here.↩︎
If you liked this post, please leave a comment.
]]>This is the second post in a series of posts:
Discuss this post on r/haskell.
Sudoku is a number placement puzzle. It consists of a 9x9 grid which is to be filled with digits from 1 to 9 such that each row, each column and each of the nine 3x3 sub-grids contain all the digits. Some of the cells of the grid come pre-filled and the player has to fill the rest.
In the previous post, we implemented a simple Sudoku solver without paying much attention to its performance characteristics. We ran1 some of 17-clue puzzles2 through our program to see how fast it was:
$ head -n100 sudoku17.txt | time stack exec sudoku
... output omitted ...
116.70 real 198.09 user 94.46 sys
So, it took about 117 seconds to solve one hundred puzzles. At this speed, it would take about 16 hours to solve all the 49151 puzzles contained in the file. This is way too slow. We need to find ways to make it faster. Let’s go back to the drawing board.
In a Sudoku puzzle, we have a partially filled 9x9 grid which we have to fill completely while following the constraints of the game.
+-------+-------+-------+
| . . . | . . . | . 1 . |
| 4 . . | . . . | . . . |
| . 2 . | . . . | . . . |
+-------+-------+-------+
| . . . | . 5 . | 4 . 7 |
| . . 8 | . . . | 3 . . |
| . . 1 | . 9 . | . . . |
+-------+-------+-------+
| 3 . . | 4 . . | 2 . . |
| . 5 . | 1 . . | . . . |
| . . . | 8 . 6 | . . . |
+-------+-------+-------+
A sample puzzle
+-------+-------+-------+
| 6 9 3 | 7 8 4 | 5 1 2 |
| 4 8 7 | 5 1 2 | 9 3 6 |
| 1 2 5 | 9 6 3 | 8 7 4 |
+-------+-------+-------+
| 9 3 2 | 6 5 1 | 4 8 7 |
| 5 6 8 | 2 4 7 | 3 9 1 |
| 7 4 1 | 3 9 8 | 6 2 5 |
+-------+-------+-------+
| 3 1 9 | 4 7 5 | 2 6 8 |
| 8 5 6 | 1 2 9 | 7 4 3 |
| 2 7 4 | 8 3 6 | 1 5 9 |
+-------+-------+-------+
and its solution
Earlier, we followed a simple pruning algorithm which removed all the solved (or fixed) digits from neighbours of the fixed cells. We repeated the pruning till the fixed and non-fixed values in the grid stopped changing (or the grid settled). Here’s an example of a grid before pruning:
+-------------------------------------+-------------------------------------+-------------------------------------+
| [123456789] [123456789] [123456789] | [123456789] [123456789] [123456789] | [123456789] 1 [123456789] |
| 4 [123456789] [123456789] | [123456789] [123456789] [123456789] | [123456789] [123456789] [123456789] |
| [123456789] 2 [123456789] | [123456789] [123456789] [123456789] | [123456789] [123456789] [123456789] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| [123456789] [123456789] [123456789] | [123456789] 5 [123456789] | 4 [123456789] 7 |
| [123456789] [123456789] 8 | [123456789] [123456789] [123456789] | 3 [123456789] [123456789] |
| [123456789] [123456789] 1 | [123456789] 9 [123456789] | [123456789] [123456789] [123456789] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| 3 [123456789] [123456789] | 4 [123456789] [123456789] | 2 [123456789] [123456789] |
| [123456789] 5 [123456789] | 1 [123456789] [123456789] | [123456789] [123456789] [123456789] |
| [123456789] [123456789] [123456789] | 8 [123456789] 6 | [123456789] [123456789] [123456789] |
+-------------------------------------+-------------------------------------+-------------------------------------+
And here’s the same grid when it settles after repeated pruning:
+-------------------------------------+-------------------------------------+-------------------------------------+
| [ 56789] [ 3 6789] [ 3 567 9] | [ 23 567 9] [ 234 6 8 ] [ 2345 789] | [ 56789] 1 [ 23456 89] |
| 4 [1 3 6789] [ 3 567 9] | [ 23 567 9] [123 6 8 ] [123 5 789] | [ 56789] [ 23 56789] [ 23 56 89] |
| [1 56789] 2 [ 3 567 9] | [ 3 567 9] [1 34 6 8 ] [1 345 789] | [ 56789] [ 3456789] [ 3456 89] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| [ 2 6 9] [ 3 6 9] [ 23 6 9] | [ 23 6 ] 5 [123 8 ] | 4 [ 2 6 89] 7 |
| [ 2 567 9] [ 4 67 9] 8 | [ 2 67 ] [12 4 6 ] [12 4 7 ] | 3 [ 2 56 9] [12 56 9] |
| [ 2 567 ] [ 34 67 ] 1 | [ 23 67 ] 9 [ 234 78 ] | [ 56 8 ] [ 2 56 8 ] [ 2 56 8 ] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| 3 [1 6 89] [ 6 9] | 4 7 [ 5 9] | 2 [ 56 89] [1 56 89] |
| [ 2 6789] 5 [ 2 4 67 9] | 1 [ 23 ] [ 23 9] | [ 6789] [ 34 6789] [ 34 6 89] |
| [12 7 9] [1 4 7 9] [ 2 4 7 9] | 8 [ 23 ] 6 | [1 5 7 9] [ 345 7 9] [1 345 9] |
+-------------------------------------+-------------------------------------+-------------------------------------+
We see how the possibilities conflicting with the fixed values are removed. We also see how some of the non-fixed cells turn into fixed ones as all their other possible values get eliminated.
This simple strategy follows directly from the constraints of Sudoku. But, are there more complex strategies which are implied indirectly?
Let’s have a look at this sample row captured from a solution in progress:
+-------------------------------------+-------------------------------------+-------------------------------------+
| 4 [ 2 6 89] 7 | 3 [ 2 56 9] [12 56 9] | [ 56 8 ] [ 2 56 8 ] [ 2 56 8 ] |
+-------------------------------------+-------------------------------------+-------------------------------------+
Notice how the sixth cell is the only one with 1
as a possibility in it. It is obvious that we should fix the sixth cell to 1
as we cannot place 1
in any other cell in the row. Let’s call this the Singles3 scenario.
But, our current solution will not fix the sixth cell to 1
till one of these cases arise:
nextGrids
function and 1
is chosen as the value to fix.This may take very long and lead to a longer solution time. Let’s assume that we recognize the Singles scenario while pruning cells and fix the cell to 1
right then. That would cut down the search tree by a lot and make the solution much faster.
It turns out, we can generalize this pattern. Let’s check out this sample row from middle of a solution:
+-------------------------------------+-------------------------------------+-------------------------------------+
| [1 4 9] 3 [1 4567 9] | [1 4 89] [1 4 6 89] [1 4 6 89] | [1 4 89] 2 [1 456789] |
+-------------------------------------+-------------------------------------+-------------------------------------+
It is a bit difficult to notice with the naked eye but there’s something special here too. The digits 5
and 7
occur only in the third and the ninth cells. Though they are accompanied by other digits in those cells, they are not present in any other cells. This means, we can place 5
and 7
either in the third or the ninth cell and no other cells. This implies that we can prune the third and ninth cells to have only 5
and 7
like this:
+-------------------------------------+-------------------------------------+-------------------------------------+
| [1 4 9] 3 [ 5 7 ] | [1 4 89] [1 4 6 89] [1 4 6 89] | [1 4 89] 2 [ 5 7 ] |
+-------------------------------------+-------------------------------------+-------------------------------------+
This is the Twins scenario. As we can imagine, this pattern extends to groups of three digits and beyond. When three digits can be found only in three cells in a block, it is the Triplets scenario, as in the example below:
+-------------------------------------+-------------------------------------+-------------------------------------+
| [ 45 7 ] [ 45 7 ] [ 5 7 ] | 2 [ 3 5 89] 6 | 1 [ 34 89] [ 34 89] |
+-------------------------------------+-------------------------------------+-------------------------------------+
In this case, the triplet digits are 3
, 8
and 9
. And as before, we can prune the block by fixing these digits in their cells:
+-------------------------------------+-------------------------------------+-------------------------------------+
| [ 45 7 ] [ 45 7 ] [ 5 7 ] | 2 [ 3 89] 6 | 1 [ 3 89] [ 3 89] |
+-------------------------------------+-------------------------------------+-------------------------------------+
Let’s call these three scenarios Exclusives in general.
We can extend this to Quadruplets scenario and further. But such scenarios occur rarely in a 9x9 Sudoku puzzle. Trying to find them may end up being more computationally expensive than the benefit we may get in solution time speedup by finding them.
Now that we have discovered these new strategies to prune cells, let’s implement them in Haskell.
We can implement the three new strategies to prune cells as one function for each. However, we can actually implement all these strategies in a single function. But, this function is a bit more complex than the previous pruning function. So first, let’s try to understand its working using tables. Let’s take this sample row:
+-------------------------------------+-------------------------------------+-------------------------------------+
| [ 4 6 9] 1 5 | [ 6 9] 7 [ 23 6 89] | [ 6 9] [ 23 6 89] [ 23 6 89] |
+-------------------------------------+-------------------------------------+-------------------------------------+
First, we make a table mapping the digits to the cells in which they occur, excluding the fixed cells:
Digit | Cells |
---|---|
2 | 6, 8, 9 |
3 | 6, 8, 9 |
4 | 1 |
6 | 1, 4, 6, 7, 8, 9 |
8 | 6, 8, 9 |
9 | 1, 4, 6, 7, 8, 9 |
Then, we flip this table and collect all the digits that occur in the same set of cells:
Cells | Digits |
---|---|
1 | 4 |
6, 8, 9 | 2, 3, 8 |
1, 4, 6, 7, 8, 9 | 6, 9 |
And finally, we remove the rows of the table in which the count of the cells is not the same as the count of the digits:
Cells | Digits |
---|---|
1 | 4 |
6, 8, 9 | 2, 3, 8 |
Voilà! We have found a Single 4
and a set of Triplets 2
, 3
and 8
. You can go over the puzzle row and verify that this indeed is the case.
Translating this logic to Haskell is quite easy now:
isPossible :: Cell -> Bool
isPossible (Possible _) = True
isPossible _ = False
exclusivePossibilities :: [Cell] -> [[Int]]
exclusivePossibilities row =
-- input
row
-- [Possible [4,6,9], Fixed 1, Fixed 5, Possible [6,9], Fixed 7, Possible [2,3,6,8,9],
-- Possible [6,9], Possible [2,3,6,8,9], Possible [2,3,6,8,9]]
-- step 1
& zip [1..9]
-- [(1,Possible [4,6,9]),(2,Fixed 1),(3,Fixed 5),(4,Possible [6,9]),(5,Fixed 7),
-- (6,Possible [2,3,6,8,9]),(7,Possible [6,9]),(8,Possible [2,3,6,8,9]),
-- (9,Possible [2,3,6,8,9])]
-- step 2
& filter (isPossible . snd)
-- [(1,Possible [4,6,9]),(4,Possible [6,9]),(6,Possible [2,3,6,8,9]),
-- (7,Possible [6,9]), (8,Possible [2,3,6,8,9]),(9,Possible [2,3,6,8,9])]
-- step 3
& Data.List.foldl'
(\acc ~(i, Possible xs) ->
Data.List.foldl' (\acc' x -> Map.insertWith prepend x [i] acc') acc xs)
Map.empty
-- fromList [(2,[9,8,6]),(3,[9,8,6]),(4,[1]),(6,[9,8,7,6,4,1]),(8,[9,8,6]),
-- (9,[9,8,7,6,4,1])]
-- step 4
& Map.filter ((< 4) . length)
-- fromList [(2,[9,8,6]),(3,[9,8,6]),(4,[1]),(8,[9,8,6])]
-- step 5
& Map.foldlWithKey'(\acc x is -> Map.insertWith prepend is [x] acc) Map.empty
-- fromList [([1],[4]),([9,8,6],[8,3,2])]
-- step 6
& Map.filterWithKey (\is xs -> length is == length xs)
-- fromList [([1],[4]),([9,8,6],[8,3,2])]
-- step 7
& Map.elems
-- [[4],[8,3,2]]
where
prepend ~[y] ys = y:ys
We extract the isPossible
function to the top level from the nextGrids
function for reuse. Then we write the exclusivePossibilities
function which finds the Exclusives in the input row. This function is written using the reverse application operator (&)
4 instead of the usual ($)
operator so that we can read it from top to bottom. We also show the intermediate values for a sample input after every step in the function chain.
The nub of the function lies in step 3 (pun intended). We do a nested fold over all the non-fixed cells and all the possible digits in them to compute the map5 which represents the first table. Thereafter, we filter the map to keep only the entries with length less than four (step 4). Then we flip it to create a new map which represents the second table (step 5). Finally, we filter the flipped map for the entries where the cell count is same as the digit count (step 6) to arrive at the final table. The step 7 just gets the values in the map which is the list of all the Exclusives in the input row.
To start with, we extract some reusable code from the previous pruneCells
function and rename it to pruneCellsByFixed
:
makeCell :: [Int] -> Maybe Cell
makeCell ys = case ys of
[] -> Nothing
[y] -> Just $ Fixed y
_ -> Just $ Possible ys
pruneCellsByFixed :: [Cell] -> Maybe [Cell]
pruneCellsByFixed cells = traverse pruneCell cells
where
fixeds = [x | Fixed x <- cells]
pruneCell (Possible xs) = makeCell (xs Data.List.\\ fixeds)
pruneCell x = Just x
Now we write the pruneCellsByExclusives
function which uses the exclusivePossibilities
function to prune the cells:
pruneCellsByExclusives :: [Cell] -> Maybe [Cell]
pruneCellsByExclusives cells = case exclusives of
[] -> Just cells
_ -> traverse pruneCell cells
where
exclusives = exclusivePossibilities cells
allExclusives = concat exclusives
pruneCell cell@(Fixed _) = Just cell
pruneCell cell@(Possible xs)
| intersection `elem` exclusives = makeCell intersection
| otherwise = Just cell
where
intersection = xs `Data.List.intersect` allExclusives
pruneCellsByExclusives
works exactly as shown in the examples above. We first find the list of Exclusives in the given cells. If there are no Exclusives, there’s nothing to do and we just return the cells. If we find any Exclusives, we traverse
the cells, pruning each cell to only the intersection of the possible digits in the cell and Exclusive digits. That’s it! We reuse the makeCell
function to create a new cell with the intersection.
As the final step, we rewrite the pruneCells
function by combining both the functions.
fixM :: (Eq t, Monad m) => (t -> m t) -> t -> m t
fixM f x = f x >>= \x' -> if x' == x then return x else fixM f x'
pruneCells :: [Cell] -> Maybe [Cell]
pruneCells cells = fixM pruneCellsByFixed cells >>= fixM pruneCellsByExclusives
We have extracted fixM
as a top level function from the pruneGrid
function. Just like the pruneGrid'
function, we need to use monadic bind (>>=
) to chain the two pruning steps. We also use fixM
to apply each step repeatedly till the pruned cells settle6.
No further code changes are required. It is time to check out the improvements.
Let’s build the program and run the exact same number of puzzles as before:
$ head -n100 sudoku17.txt | time stack exec sudoku
... output omitted ...
0.53 real 0.58 user 0.23 sys
Woah! It is way faster than before. Let’s solve all the puzzles now:
$ cat sudoku17.txt | time stack exec sudoku > /dev/null
282.98 real 407.25 user 109.27 sys
So it is took about 283 seconds to solve all the 49151 puzzles. The speedup is about 200x7. That’s about 5.8 milliseconds per puzzle.
Let’s do a quick profiling to see where the time is going:
$ stack build --profile
$ head -n1000 sudoku17.txt | stack exec -- sudoku +RTS -p > /dev/null
This generates a file named sudoku.prof
with the profiling results. Here are the top five most time-taking functions (cleaned for brevity):
Cost Center | Source | %time | %alloc |
---|---|---|---|
exclusivePossibilities |
(49,1)-(62,26) | 17.6 | 11.4 |
pruneCellsByFixed.pruneCell |
(75,5)-(76,36) | 16.9 | 30.8 |
exclusivePossibilities.\.\ |
55:38-70 | 12.2 | 20.3 |
fixM.\ |
13:27-65 | 10.0 | 0.0 |
== |
15:56-57 | 7.2 | 0.0 |
Looking at the report, my guess is that a lot of time is going into list operations. Lists are known to be inefficient in Haskell so maybe we should switch to some other data structures?
As per the comment below by Chris Casinghino, I ran both the versions of code without the -threaded
, -rtsopts
and -with-rtsopts=-N
options. The time for previous post’s code:
$ head -n100 sudoku17.txt | time stack exec sudoku
... output omitted ...
96.54 real 95.90 user 0.66 sys
And the time for this post’s code:
$ cat sudoku17.txt | time stack exec sudoku > /dev/null
258.97 real 257.34 user 1.52 sys
So, both the versions run about 10% faster without the threading options. I suspect this has something to do with GHC’s parallel GC as described in this post. So for now, I’ll keep threading disabled.
In this post, we improved upon our simple Sudoku solution from the last time. We discovered and implemented a new strategy to prune cells, and we achieved a 200x speedup. But profiling shows that we still have many possibilities for improvements. We’ll work on that and more in the upcoming posts in this series. The code till now is available here. Discuss this post on r/haskell or leave a comment.
All the runs were done on my MacBook Pro from 2014 with 2.2 GHz Intel Core i7 CPU and 16 GB memory.↩︎
At least 17 cells must be pre-filled in a Sudoku puzzle for it to have a unique solution. So 17-clue puzzles are the most difficult of all puzzles. This paper by McGuire, Tugemann and Civario gives the proof of the same.↩︎
“Single” as in “Single child”↩︎
Reverse application operation is not used much in Haskell. But it is the preferred way of function chaining in some other functional programming languages like Clojure, FSharp, and Elixir.↩︎
We use Data.Map.Strict as the map implementation.↩︎
We need to run pruneCellsByFixed
and pruneCellsByExclusives
repeatedly using fixM
because an unsettled row can lead to wrong solutions.
Imagine a row which just got a 9
fixed because of pruneCellsByFixed
. If we don’t run the function again, the row may be left with one non-fixed cell with a 9
. When we run this row through pruneCellsByExclusives
, it’ll consider the 9
in the non-fixed cell as a Single and fix it. This will lead to two 9
s in the same row, causing the solution to fail.↩︎
Speedup calculation: 116.7 / 100 * 49151 / 282.98 = 202.7↩︎
If you liked this post, please leave a comment.
]]>Haskell is a purely functional programming language. It is a good choice to solve Sudoku given the problem’s combinatorial nature. The aim of this series of posts is to write a fast Sudoku solver in Haskell. We’ll focus on both implementing the solution and making it efficient, step-by-step, starting with a slow but simple solution in this post1.
This is the first post in a series of posts:
Discuss this post on r/haskell.
Solving Sudoku is a constraint satisfaction problem. We are given a partially filled grid which we have to fill completely such that each of the following constraints are satisfied:
+-------+-------+-------+
| . . . | . . . | . 1 . |
| 4 . . | . . . | . . . |
| . 2 . | . . . | . . . |
+-------+-------+-------+
| . . . | . 5 . | 4 . 7 |
| . . 8 | . . . | 3 . . |
| . . 1 | . 9 . | . . . |
+-------+-------+-------+
| 3 . . | 4 . . | 2 . . |
| . 5 . | 1 . . | . . . |
| . . . | 8 . 6 | . . . |
+-------+-------+-------+
A sample puzzle
+-------+-------+-------+
| 6 9 3 | 7 8 4 | 5 1 2 |
| 4 8 7 | 5 1 2 | 9 3 6 |
| 1 2 5 | 9 6 3 | 8 7 4 |
+-------+-------+-------+
| 9 3 2 | 6 5 1 | 4 8 7 |
| 5 6 8 | 2 4 7 | 3 9 1 |
| 7 4 1 | 3 9 8 | 6 2 5 |
+-------+-------+-------+
| 3 1 9 | 4 7 5 | 2 6 8 |
| 8 5 6 | 1 2 9 | 7 4 3 |
| 2 7 4 | 8 3 6 | 1 5 9 |
+-------+-------+-------+
and its solution
Each cell in the grid is member of one row, one column and one sub-grid (called block in general). Digits in the pre-filled cells impose constraints on the rows, columns, and sub-grids they are part of. For example, if a cell contains 1
then no other cell in that cell’s row, column or sub-grid can contain 1
. Given these constraints, we can devise a simple algorithm to solve Sudoku:
+-------------------------------------+-------------------------------------+-------------------------------------+
| [123456789] [123456789] [123456789] | [123456789] [123456789] [123456789] | [123456789] 1 [123456789] |
| 4 [123456789] [123456789] | [123456789] [123456789] [123456789] | [123456789] [123456789] [123456789] |
| [123456789] 2 [123456789] | [123456789] [123456789] [123456789] | [123456789] [123456789] [123456789] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| [123456789] [123456789] [123456789] | [123456789] 5 [123456789] | 4 [123456789] 7 |
| [123456789] [123456789] 8 | [123456789] [123456789] [123456789] | 3 [123456789] [123456789] |
| [123456789] [123456789] 1 | [123456789] 9 [123456789] | [123456789] [123456789] [123456789] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| 3 [123456789] [123456789] | 4 [123456789] [123456789] | 2 [123456789] [123456789] |
| [123456789] 5 [123456789] | 1 [123456789] [123456789] | [123456789] [123456789] [123456789] |
| [123456789] [123456789] [123456789] | 8 [123456789] 6 | [123456789] [123456789] [123456789] |
+-------------------------------------+-------------------------------------+-------------------------------------+
4
of the row-2-column-1 cell from its neighboring cells:+-------------------------------------+-------------------------------------+-------------------------------------+
| [123 56789] [123 56789] [123 56789] | [123456789] [123456789] [123456789] | [123456789] 1 [123456789] |
| 4 [123 56789] [123 56789] | [123 56789] [123 56789] [123 56789] | [123 56789] [123 56789] [123 56789] |
| [123 56789] 2 [123 56789] | [123456789] [123456789] [123456789] | [123456789] [123456789] [123456789] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| [123 56789] [123456789] [123456789] | [123456789] 5 [123456789] | 4 [123456789] 7 |
| [123 56789] [123456789] 8 | [123456789] [123456789] [123456789] | 3 [123456789] [123456789] |
| [123 56789] [123456789] 1 | [123456789] 9 [123456789] | [123456789] [123456789] [123456789] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| 3 [123456789] [123456789] | 4 [123456789] [123456789] | 2 [123456789] [123456789] |
| [123 56789] 5 [123456789] | 1 [123456789] [123456789] | [123456789] [123456789] [123456789] |
| [123 56789] [123456789] [123456789] | 8 [123456789] 6 | [123456789] [123456789] [123456789] |
+-------------------------------------+-------------------------------------+-------------------------------------+
+-------------------------------------+-------------------------------------+-------------------------------------+
| [ 56789] [ 3 6789] [ 3 567 9] | [ 23 567 9] [ 234 678 ] [ 2345 789] | [ 56789] 1 [ 23456 89] |
| 4 [1 3 6789] [ 3 567 9] | [ 23 567 9] [123 678 ] [123 5 789] | [ 56789] [ 23 56789] [ 23 56 89] |
| [1 56789] 2 [ 3 567 9] | [ 3 567 9] [1 34 678 ] [1 345 789] | [ 56789] [ 3456789] [ 3456 89] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| [ 2 6 9] [ 3 6 9] [ 23 6 9] | [ 23 6 ] 5 [123 8 ] | 4 [ 2 6 89] 7 |
| [ 2 567 9] [ 4 67 9] 8 | [ 2 67 ] [12 4 67 ] [12 4 7 ] | 3 [ 2 56 9] [12 56 9] |
| [ 2 567 ] [ 34 67 ] 1 | [ 23 67 ] 9 [ 234 78 ] | [ 56 8 ] [ 2 56 8 ] [ 2 56 8 ] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| 3 [1 6789] [ 67 9] | 4 7 [ 5 7 9] | 2 [ 56789] [1 56 89] |
| [ 2 6789] 5 [ 2 4 67 9] | 1 [ 23 7 ] [ 23 7 9] | [ 6789] [ 34 6789] [ 34 6 89] |
| [12 7 9] [1 4 7 9] [ 2 4 7 9] | 8 [ 23 7 ] 6 | [1 5 7 9] [ 345 7 9] [1 345 9] |
+-------------------------------------+-------------------------------------+-------------------------------------+
+-------------------------------------+-------------------------------------+-------------------------------------+
| [ 56789] [ 3 6789] [ 3 567 9] | [ 23 567 9] [ 234 6 8 ] [ 2345 789] | [ 56789] 1 [ 23456 89] |
| 4 [1 3 6789] [ 3 567 9] | [ 23 567 9] [123 6 8 ] [123 5 789] | [ 56789] [ 23 56789] [ 23 56 89] |
| [1 56789] 2 [ 3 567 9] | [ 3 567 9] [1 34 6 8 ] [1 345 789] | [ 56789] [ 3456789] [ 3456 89] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| [ 2 6 9] [ 3 6 9] [ 23 6 9] | [ 23 6 ] 5 [123 8 ] | 4 [ 2 6 89] 7 |
| [ 2 567 9] [ 4 67 9] 8 | [ 2 67 ] [12 4 6 ] [12 4 7 ] | 3 [ 2 56 9] [12 56 9] |
| [ 2 567 ] [ 34 67 ] 1 | [ 23 67 ] 9 [ 234 78 ] | [ 56 8 ] [ 2 56 8 ] [ 2 56 8 ] |
+-------------------------------------+-------------------------------------+-------------------------------------+
| 3 [1 6 89] [ 6 9] | 4 7 [ 5 9] | 2 [ 56 89] [1 56 89] |
| [ 2 6789] 5 [ 2 4 67 9] | 1 [ 23 ] [ 23 9] | [ 6789] [ 34 6789] [ 34 6 89] |
| [12 7 9] [1 4 7 9] [ 2 4 7 9] | 8 [ 23 ] 6 | [1 5 7 9] [ 345 7 9] [1 345 9] |
+-------------------------------------+-------------------------------------+-------------------------------------+
This algorithm is actually a Depth-First Search on the state space of the grid configurations. It guarantees to either find a solution or prove a puzzle to be unsolvable.
We start with writing types to represent the cells and the grid:
A cell is either fixed with a particular digit or has a set of digits as possibilities. So it is natural to represent it as a sum type with Fixed
and Possible
constructors. A row is a list of cells and a grid is a list of rows.
We’ll take the input puzzle as a string of 81 characters representing the cells, left-to-right and top-to-bottom. An example is:
.......1.4.........2...........5.4.7..8...3....1.9....3..4..2...5.1........8.6...
Here, .
represents an non-filled cell. Let’s write a function to read this input and parse it to our Grid
data structure:
readGrid :: String -> Maybe Grid
readGrid s
| length s == 81 = traverse (traverse readCell) . Data.List.Split.chunksOf 9 $ s
| otherwise = Nothing
where
readCell '.' = Just $ Possible [1..9]
readCell c
| Data.Char.isDigit c && c > '0' = Just . Fixed . Data.Char.digitToInt $ c
| otherwise = Nothing
readGrid
return a Just grid
if the input is correct, else it returns a Nothing
. It parses a .
to a Possible
cell with all digits as possibilities, and a digit char to a Fixed
cell with that digit. Let’s try it out in the REPL:
*Main> Just grid = readGrid ".......1.4.........2...........5.4.7..8...3....1.9....3..4..2...5.1........8.6..."
*Main> mapM_ print grid
[Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Fixed 1,Possible [1,2,3,4,5,6,7,8,9]]
[Fixed 4,Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9]]
[Possible [1,2,3,4,5,6,7,8,9],Fixed 2,Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9]]
[Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Fixed 5,Possible [1,2,3,4,5,6,7,8,9],Fixed 4,Possible [1,2,3,4,5,6,7,8,9],Fixed 7]
[Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Fixed 8,Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Fixed 3,Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9]]
[Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Fixed 1,Possible [1,2,3,4,5,6,7,8,9],Fixed 9,Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9]]
[Fixed 3,Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Fixed 4,Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Fixed 2,Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9]]
[Possible [1,2,3,4,5,6,7,8,9],Fixed 5,Possible [1,2,3,4,5,6,7,8,9],Fixed 1,Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9]]
[Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Fixed 8,Possible [1,2,3,4,5,6,7,8,9],Fixed 6,Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9],Possible [1,2,3,4,5,6,7,8,9]]
The output is a bit unreadable but correct. We can write a few functions to clean it up:
showGrid :: Grid -> String
showGrid = unlines . map (unwords . map showCell)
where
showCell (Fixed x) = show x
showCell _ = "."
showGridWithPossibilities :: Grid -> String
showGridWithPossibilities = unlines . map (unwords . map showCell)
where
showCell (Fixed x) = show x ++ " "
showCell (Possible xs) =
(++ "]")
. Data.List.foldl' (\acc x -> acc ++ if x `elem` xs then show x else " ") "["
$ [1..9]
Back to the REPL again:
*Main> Just grid = readGrid ".......1.4.........2...........5.4.7..8...3....1.9....3..4..2...5.1........8.6..."
*Main> putStrLn $ showGrid grid
. . . . . . . 1 .
4 . . . . . . . .
. 2 . . . . . . .
. . . . 5 . 4 . 7
. . 8 . . . 3 . .
. . 1 . 9 . . . .
3 . . 4 . . 2 . .
. 5 . 1 . . . . .
. . . 8 . 6 . . .
*Main> putStrLn $ showGridWithPossibilities grid
[123456789] [123456789] [123456789] [123456789] [123456789] [123456789] [123456789] 1 [123456789]
4 [123456789] [123456789] [123456789] [123456789] [123456789] [123456789] [123456789] [123456789]
[123456789] 2 [123456789] [123456789] [123456789] [123456789] [123456789] [123456789] [123456789]
[123456789] [123456789] [123456789] [123456789] 5 [123456789] 4 [123456789] 7
[123456789] [123456789] 8 [123456789] [123456789] [123456789] 3 [123456789] [123456789]
[123456789] [123456789] 1 [123456789] 9 [123456789] [123456789] [123456789] [123456789]
3 [123456789] [123456789] 4 [123456789] [123456789] 2 [123456789] [123456789]
[123456789] 5 [123456789] 1 [123456789] [123456789] [123456789] [123456789] [123456789]
[123456789] [123456789] [123456789] 8 [123456789] 6 [123456789] [123456789] [123456789]
The output is more readable now. We see that, at the start, all the non-filled cells have all the digits as possible values. We’ll use these functions for debugging as we go forward. We can now start solving the puzzle.
We can remove the digits of fixed cells from their neighboring cells, one cell as a time. But, it is faster to find all the fixed digits in a row of cells and remove them from the possibilities of all the non-fixed cells of the row, at once. Then we can repeat this pruning step for all the rows of the grid (and columns and sub-grids too! We’ll see how).
pruneCells :: [Cell] -> Maybe [Cell]
pruneCells cells = traverse pruneCell cells
where
fixeds = [x | Fixed x <- cells]
pruneCell (Possible xs) = case xs Data.List.\\ fixeds of
[] -> Nothing
[y] -> Just $ Fixed y
ys -> Just $ Possible ys
pruneCell x = Just x
pruneCells
prunes a list of cells as described before. We start with finding the fixed digits in the list of cells. Then we go over each non-fixed cells, removing the fixed digits we found, from their possible values. Two special cases arise:
Nothing
in that case.We use the traverse
function for pruning the cells so that a Nothing
resulting from pruning one cell propagates to the entire list.
Let’s take it for a spin in the REPL:
*Main> Just grid = readGrid "6......1.4.........2...........5.4.7..8...3....1.9....3..4..2...5.1........8.6..."
*Main> putStr $ showGridWithPossibilities $ [head grid] -- first row of the grid
6 [123456789] [123456789] [123456789] [123456789] [123456789] [123456789] 1 [123456789]
*Main> putStr $ showGridWithPossibilities [fromJust $ pruneCells $ head grid] -- same row after pruning
6 [ 2345 789] [ 2345 789] [ 2345 789] [ 2345 789] [ 2345 789] [ 2345 789] 1 [ 2345 789]
It works! 6
and 1
are removed from the possibilities of the other cells. Now we are ready for …
Pruning a grid requires us to prune each row, each column and each sub-grid. Let’s try to solve it in the REPL first:
*Main> Just grid = readGrid "6......1.4.........2...........5.4.7..8...3....1.9....3..4..2...5.1........8.6..."
*Main> Just grid' = traverse pruneCells grid
*Main> putStr $ showGridWithPossibilities grid'
6 [ 2345 789] [ 2345 789] [ 2345 789] [ 2345 789] [ 2345 789] [ 2345 789] 1 [ 2345 789]
4 [123 56789] [123 56789] [123 56789] [123 56789] [123 56789] [123 56789] [123 56789] [123 56789]
[1 3456789] 2 [1 3456789] [1 3456789] [1 3456789] [1 3456789] [1 3456789] [1 3456789] [1 3456789]
[123 6 89] [123 6 89] [123 6 89] [123 6 89] 5 [123 6 89] 4 [123 6 89] 7
[12 4567 9] [12 4567 9] 8 [12 4567 9] [12 4567 9] [12 4567 9] 3 [12 4567 9] [12 4567 9]
[ 2345678 ] [ 2345678 ] 1 [ 2345678 ] 9 [ 2345678 ] [ 2345678 ] [ 2345678 ] [ 2345678 ]
3 [1 56789] [1 56789] 4 [1 56789] [1 56789] 2 [1 56789] [1 56789]
[ 234 6789] 5 [ 234 6789] 1 [ 234 6789] [ 234 6789] [ 234 6789] [ 234 6789] [ 234 6789]
[12345 7 9] [12345 7 9] [12345 7 9] 8 [12345 7 9] 6 [12345 7 9] [12345 7 9] [12345 7 9]
By traverse
-ing the grid with pruneCells
, we are able to prune each row, one-by-one. Since pruning a row doesn’t affect another row, we don’t have to pass the resulting rows between each pruning step. That is to say, traverse
is enough for us, we don’t need foldl
here.
How do we do the same thing for columns now? Since our representation for the grid is rows-first, we first need to convert it to a columns-first representation. Luckily, that’s what Data.List.transpose
function does:
*Main> Just grid = readGrid "693784512487512936125963874932651487568247391741398625319475268856129743274836159"
*Main> putStr $ showGrid grid
6 9 3 7 8 4 5 1 2
4 8 7 5 1 2 9 3 6
1 2 5 9 6 3 8 7 4
9 3 2 6 5 1 4 8 7
5 6 8 2 4 7 3 9 1
7 4 1 3 9 8 6 2 5
3 1 9 4 7 5 2 6 8
8 5 6 1 2 9 7 4 3
2 7 4 8 3 6 1 5 9
*Main> putStr $ showGrid $ Data.List.transpose grid
6 4 1 9 5 7 3 8 2
9 8 2 3 6 4 1 5 7
3 7 5 2 8 1 9 6 4
7 5 9 6 2 3 4 1 8
8 1 6 5 4 9 7 2 3
4 2 3 1 7 8 5 9 6
5 9 8 4 3 6 2 7 1
1 3 7 8 9 2 6 4 5
2 6 4 7 1 5 8 3 9
Pruning columns is easy now:
*Main> Just grid = readGrid "6......1.4.........2...........5.4.7..8...3....1.9....3..4..2...5.1........8.6..."
*Main> Just grid' = fmap Data.List.transpose . traverse pruneCells . Data.List.transpose $ grid
*Main> putStr $ showGridWithPossibilities grid'
6 [1 34 6789] [ 234567 9] [ 23 567 9] [1234 678 ] [12345 789] [1 56789] 1 [123456 89]
4 [1 34 6789] [ 234567 9] [ 23 567 9] [1234 678 ] [12345 789] [1 56789] [ 23456789] [123456 89]
[12 5 789] 2 [ 234567 9] [ 23 567 9] [1234 678 ] [12345 789] [1 56789] [ 23456789] [123456 89]
[12 5 789] [1 34 6789] [ 234567 9] [ 23 567 9] 5 [12345 789] 4 [ 23456789] 7
[12 5 789] [1 34 6789] 8 [ 23 567 9] [1234 678 ] [12345 789] 3 [ 23456789] [123456 89]
[12 5 789] [1 34 6789] 1 [ 23 567 9] 9 [12345 789] [1 56789] [ 23456789] [123456 89]
3 [1 34 6789] [ 234567 9] 4 [1234 678 ] [12345 789] 2 [ 23456789] [123456 89]
[12 5 789] 5 [ 234567 9] 1 [1234 678 ] [12345 789] [1 56789] [ 23456789] [123456 89]
[12 5 789] [1 34 6789] [ 234567 9] 8 [1234 678 ] 6 [1 56789] [ 23456789] [123456 89]
First, we transpose
the grid to convert the columns into rows. Then, we prune the rows by traverse
-ing pruneCells
over them. And finally, we turn the rows back into columns by transpose
-ing the grid back again. The last transpose
needs to be fmap
-ped because traverse pruneCells
returns a Maybe
.
Pruning sub-grids is a bit trickier. Following the same idea as pruning columns, we need two functions to transform the sub-grids into rows and back. Let’s write the first one:
subGridsToRows :: Grid -> Grid
subGridsToRows =
concatMap (\rows -> let [r1, r2, r3] = map (Data.List.Split.chunksOf 3) rows
in zipWith3 (\a b c -> a ++ b ++ c) r1 r2 r3)
. Data.List.Split.chunksOf 3
And try it out:
*Main> Just grid = readGrid "693784512487512936125963874932651487568247391741398625319475268856129743274836159"
*Main> putStr $ showGrid grid
6 9 3 7 8 4 5 1 2
4 8 7 5 1 2 9 3 6
1 2 5 9 6 3 8 7 4
9 3 2 6 5 1 4 8 7
5 6 8 2 4 7 3 9 1
7 4 1 3 9 8 6 2 5
3 1 9 4 7 5 2 6 8
8 5 6 1 2 9 7 4 3
2 7 4 8 3 6 1 5 9
*Main> putStr $ showGrid $ subGridsToRows grid
6 9 3 4 8 7 1 2 5
7 8 4 5 1 2 9 6 3
5 1 2 9 3 6 8 7 4
9 3 2 5 6 8 7 4 1
6 5 1 2 4 7 3 9 8
4 8 7 3 9 1 6 2 5
3 1 9 8 5 6 2 7 4
4 7 5 1 2 9 8 3 6
2 6 8 7 4 3 1 5 9
You can go over the code and the output and make yourself sure that it works. Also, it turns out that we don’t need to write the back-transform function. subGridsToRows
is its own back-transform:
*Main> putStr $ showGrid grid
6 9 3 7 8 4 5 1 2
4 8 7 5 1 2 9 3 6
1 2 5 9 6 3 8 7 4
9 3 2 6 5 1 4 8 7
5 6 8 2 4 7 3 9 1
7 4 1 3 9 8 6 2 5
3 1 9 4 7 5 2 6 8
8 5 6 1 2 9 7 4 3
2 7 4 8 3 6 1 5 9
*Main> putStr $ showGrid $ subGridsToRows $ subGridsToRows $ grid
6 9 3 7 8 4 5 1 2
4 8 7 5 1 2 9 3 6
1 2 5 9 6 3 8 7 4
9 3 2 6 5 1 4 8 7
5 6 8 2 4 7 3 9 1
7 4 1 3 9 8 6 2 5
3 1 9 4 7 5 2 6 8
8 5 6 1 2 9 7 4 3
2 7 4 8 3 6 1 5 9
Nice! Now writing the sub-grid pruning function is easy:
*Main> Just grid = readGrid "6......1.4.........2...........5.4.7..8...3....1.9....3..4..2...5.1........8.6..."
*Main> Just grid' = fmap subGridsToRows . traverse pruneCells . subGridsToRows $ grid
*Main> putStr $ showGridWithPossibilities grid'
6 [1 3 5 789] [1 3 5 789] [123456789] [123456789] [123456789] [ 23456789] 1 [ 23456789]
4 [1 3 5 789] [1 3 5 789] [123456789] [123456789] [123456789] [ 23456789] [ 23456789] [ 23456789]
[1 3 5 789] 2 [1 3 5 789] [123456789] [123456789] [123456789] [ 23456789] [ 23456789] [ 23456789]
[ 234567 9] [ 234567 9] [ 234567 9] [1234 678 ] 5 [1234 678 ] 4 [12 56 89] 7
[ 234567 9] [ 234567 9] 8 [1234 678 ] [1234 678 ] [1234 678 ] 3 [12 56 89] [12 56 89]
[ 234567 9] [ 234567 9] 1 [1234 678 ] 9 [1234 678 ] [12 56 89] [12 56 89] [12 56 89]
3 [12 4 6789] [12 4 6789] 4 [ 23 5 7 9] [ 23 5 7 9] 2 [1 3456789] [1 3456789]
[12 4 6789] 5 [12 4 6789] 1 [ 23 5 7 9] [ 23 5 7 9] [1 3456789] [1 3456789] [1 3456789]
[12 4 6789] [12 4 6789] [12 4 6789] 8 [ 23 5 7 9] 6 [1 3456789] [1 3456789] [1 3456789]
It works well. Now we can string together these three steps to prune the entire grid. We also have to make sure that result of pruning each step is fed into the next step. This is so that the fixed cells created into one step cause more pruning in the further steps. We use monadic bind (>>=
) for that. Here’s the final code:
pruneGrid' :: Grid -> Maybe Grid
pruneGrid' grid =
traverse pruneCells grid
>>= fmap Data.List.transpose . traverse pruneCells . Data.List.transpose
>>= fmap subGridsToRows . traverse pruneCells . subGridsToRows
And the test:
*Main> Just grid = readGrid "6......1.4.........2...........5.4.7..8...3....1.9....3..4..2...5.1........8.6..."
*Main> Just grid' = pruneGrid' grid
*Main> putStr $ showGridWithPossibilities grid'
6 [ 3 789] [ 3 5 7 9] [ 23 5 7 9] [ 234 78 ] [ 2345 789] [ 5 789] 1 [ 2345 89]
4 [1 3 789] [ 3 5 7 9] [ 23 567 9] [123 678 ] [123 5 789] [ 56789] [ 23 56789] [ 23 56 89]
[1 5 789] 2 [ 3 5 7 9] [ 3 567 9] [1 34 678 ] [1 345 789] [ 56789] [ 3456789] [ 3456 89]
[ 2 9] [ 3 6 9] [ 23 6 9] [ 23 6 ] 5 [123 8 ] 4 [ 2 6 89] 7
[ 2 5 7 9] [ 4 67 9] 8 [ 2 67 ] [12 4 67 ] [12 4 7 ] 3 [ 2 56 9] [12 56 9]
[ 2 5 7 ] [ 34 67 ] 1 [ 23 67 ] 9 [ 234 78 ] [ 56 8 ] [ 2 56 8 ] [ 2 56 8 ]
3 [1 6789] [ 67 9] 4 7 [ 5 7 9] 2 [ 56789] [1 56 89]
[ 2 789] 5 [ 2 4 67 9] 1 [ 23 7 ] [ 23 7 9] [ 6789] [ 34 6789] [ 34 6 89]
[12 7 9] [1 4 7 9] [ 2 4 7 9] 8 [ 23 7 ] 6 [1 5 7 9] [ 345 7 9] [1 345 9]
*Main> putStr $ showGrid grid
6 . . . . . . 1 .
4 . . . . . . . .
. 2 . . . . . . .
. . . . 5 . 4 . 7
. . 8 . . . 3 . .
. . 1 . 9 . . . .
3 . . 4 . . 2 . .
. 5 . 1 . . . . .
. . . 8 . 6 . . .
*Main> putStr $ showGrid grid'
6 . . . . . . 1 .
4 . . . . . . . .
. 2 . . . . . . .
. . . . 5 . 4 . 7
. . 8 . . . 3 . .
. . 1 . 9 . . . .
3 . . 4 7 . 2 . .
. 5 . 1 . . . . .
. . . 8 . 6 . . .
We can clearly see the massive pruning of possibilities all around the grid. We also see a 7
pop up in the row-7-column-5 cell. This means that we can prune the grid further, until it settles. If you are familiar with Haskell, you may recognize this as trying to find a fixed point for the pruneGrid'
function, except in a monadic context. It is simple to implement:
pruneGrid :: Grid -> Maybe Grid
pruneGrid = fixM pruneGrid'
where
fixM f x = f x >>= \x' -> if x' == x then return x else fixM f x'
The crux of this code is the fixM
function. It takes a monadic function f
and an initial value, and recursively calls itself till the return value settles. Let’s do another round in the REPL:
*Main> Just grid = readGrid "6......1.4.........2...........5.4.7..8...3....1.9....3..4..2...5.1........8.6..."
*Main> Just grid' = pruneGrid grid
*Main> putStr $ showGridWithPossibilities grid'
6 [ 3 789] [ 3 5 7 9] [ 23 5 7 9] [ 234 8 ] [ 2345 789] [ 5 789] 1 [ 2345 89]
4 [1 3 789] [ 3 5 7 9] [ 23 567 9] [123 6 8 ] [123 5 789] [ 56789] [ 23 56789] [ 23 56 89]
[1 5 789] 2 [ 3 5 7 9] [ 3 567 9] [1 34 6 8 ] [1 345 789] [ 56789] [ 3456789] [ 3456 89]
[ 2 9] [ 3 6 9] [ 23 6 9] [ 23 6 ] 5 [123 8 ] 4 [ 2 6 89] 7
[ 2 5 7 9] [ 4 67 9] 8 [ 2 67 ] [12 4 6 ] [12 4 7 ] 3 [ 2 56 9] [12 56 9]
[ 2 5 7 ] [ 34 67 ] 1 [ 23 67 ] 9 [ 234 78 ] [ 56 8 ] [ 2 56 8 ] [ 2 56 8 ]
3 [1 6 89] [ 6 9] 4 7 [ 5 9] 2 [ 56 89] [1 56 89]
[ 2 789] 5 [ 2 4 67 9] 1 [ 23 ] [ 23 9] [ 6789] [ 34 6789] [ 34 6 89]
[12 7 9] [1 4 7 9] [ 2 4 7 9] 8 [ 23 ] 6 [1 5 7 9] [ 345 7 9] [1 345 9]
We see that 7
in the row-7-column-5 cell is eliminated from all its neighboring cells. We can’t prune the grid anymore. Now it is time to make the choice.
One the grid is settled, we need to choose a non-fixed cell and make it fixed by assuming one of its possible values. This gives us two grids, next in the state-space of the solution search:
We call this function, nextGrids
:
nextGrids :: Grid -> (Grid, Grid)
nextGrids grid =
let (i, first@(Fixed _), rest) =
fixCell
. Data.List.minimumBy (compare `Data.Function.on` (possibilityCount . snd))
. filter (isPossible . snd)
. zip [0..]
. concat
$ grid
in (replace2D i first grid, replace2D i rest grid)
where
isPossible (Possible _) = True
isPossible _ = False
possibilityCount (Possible xs) = length xs
possibilityCount (Fixed _) = 1
fixCell (i, Possible [x, y]) = (i, Fixed x, Fixed y)
fixCell (i, Possible (x:xs)) = (i, Fixed x, Possible xs)
fixCell _ = error "Impossible case"
replace2D :: Int -> a -> [[a]] -> [[a]]
replace2D i v =
let (x, y) = (i `quot` 9, i `mod` 9) in replace x (replace y (const v))
replace p f xs = [if i == p then f x else x | (x, i) <- zip xs [0..]]
We choose the non-fixed cell with least count of possibilities as the pivot. This strategy make sense intuitively, as with a cell with fewest possibilities, we have the most chance of being right when assuming one. Fixing a non-fixed cell leads to one of the two cases:
Then all we are left with is replacing the non-fixed cell with its fixed and fixed/non-fixed choices, which we do with some math and some list traversal. A quick check on the REPL:
*Main> Just grid = readGrid "6......1.4.........2...........5.4.7..8...3....1.9....3..4..2...5.1........8.6..."
*Main> Just grid' = pruneGrid grid
*Main> putStr $ showGridWithPossibilities grid'
6 [ 3 789] [ 3 5 7 9] [ 23 5 7 9] [ 234 8 ] [ 2345 789] [ 5 789] 1 [ 2345 89]
4 [1 3 789] [ 3 5 7 9] [ 23 567 9] [123 6 8 ] [123 5 789] [ 56789] [ 23 56789] [ 23 56 89]
[1 5 789] 2 [ 3 5 7 9] [ 3 567 9] [1 34 6 8 ] [1 345 789] [ 56789] [ 3456789] [ 3456 89]
[ 2 9] [ 3 6 9] [ 23 6 9] [ 23 6 ] 5 [123 8 ] 4 [ 2 6 89] 7
[ 2 5 7 9] [ 4 67 9] 8 [ 2 67 ] [12 4 6 ] [12 4 7 ] 3 [ 2 56 9] [12 56 9]
[ 2 5 7 ] [ 34 67 ] 1 [ 23 67 ] 9 [ 234 78 ] [ 56 8 ] [ 2 56 8 ] [ 2 56 8 ]
3 [1 6 89] [ 6 9] 4 7 [ 5 9] 2 [ 56 89] [1 56 89]
[ 2 789] 5 [ 2 4 67 9] 1 [ 23 ] [ 23 9] [ 6789] [ 34 6789] [ 34 6 89]
[12 7 9] [1 4 7 9] [ 2 4 7 9] 8 [ 23 ] 6 [1 5 7 9] [ 345 7 9] [1 345 9]
*Main> -- the row-4-column-1 cell is the first cell with only two possibilities, [2, 9].
*Main> -- it is chosen as the pivot cell to find the next grids.
*Main> (grid1, grid2) = nextGrids grid'
*Main> putStr $ showGridWithPossibilities grid1
6 [ 3 789] [ 3 5 7 9] [ 23 5 7 9] [ 234 8 ] [ 2345 789] [ 5 789] 1 [ 2345 89]
4 [1 3 789] [ 3 5 7 9] [ 23 567 9] [123 6 8 ] [123 5 789] [ 56789] [ 23 56789] [ 23 56 89]
[1 5 789] 2 [ 3 5 7 9] [ 3 567 9] [1 34 6 8 ] [1 345 789] [ 56789] [ 3456789] [ 3456 89]
2 [ 3 6 9] [ 23 6 9] [ 23 6 ] 5 [123 8 ] 4 [ 2 6 89] 7
[ 2 5 7 9] [ 4 67 9] 8 [ 2 67 ] [12 4 6 ] [12 4 7 ] 3 [ 2 56 9] [12 56 9]
[ 2 5 7 ] [ 34 67 ] 1 [ 23 67 ] 9 [ 234 78 ] [ 56 8 ] [ 2 56 8 ] [ 2 56 8 ]
3 [1 6 89] [ 6 9] 4 7 [ 5 9] 2 [ 56 89] [1 56 89]
[ 2 789] 5 [ 2 4 67 9] 1 [ 23 ] [ 23 9] [ 6789] [ 34 6789] [ 34 6 89]
[12 7 9] [1 4 7 9] [ 2 4 7 9] 8 [ 23 ] 6 [1 5 7 9] [ 345 7 9] [1 345 9]
*Main> putStr $ showGridWithPossibilities grid2
6 [ 3 789] [ 3 5 7 9] [ 23 5 7 9] [ 234 8 ] [ 2345 789] [ 5 789] 1 [ 2345 89]
4 [1 3 789] [ 3 5 7 9] [ 23 567 9] [123 6 8 ] [123 5 789] [ 56789] [ 23 56789] [ 23 56 89]
[1 5 789] 2 [ 3 5 7 9] [ 3 567 9] [1 34 6 8 ] [1 345 789] [ 56789] [ 3456789] [ 3456 89]
9 [ 3 6 9] [ 23 6 9] [ 23 6 ] 5 [123 8 ] 4 [ 2 6 89] 7
[ 2 5 7 9] [ 4 67 9] 8 [ 2 67 ] [12 4 6 ] [12 4 7 ] 3 [ 2 56 9] [12 56 9]
[ 2 5 7 ] [ 34 67 ] 1 [ 23 67 ] 9 [ 234 78 ] [ 56 8 ] [ 2 56 8 ] [ 2 56 8 ]
3 [1 6 89] [ 6 9] 4 7 [ 5 9] 2 [ 56 89] [1 56 89]
[ 2 789] 5 [ 2 4 67 9] 1 [ 23 ] [ 23 9] [ 6789] [ 34 6789] [ 34 6 89]
[12 7 9] [1 4 7 9] [ 2 4 7 9] 8 [ 23 ] 6 [1 5 7 9] [ 345 7 9] [1 345 9]
We have implemented parts of our algorithm till now. Now we’ll put everything together to solve the puzzle. First, we need to know if we are done or have messed up:
isGridFilled :: Grid -> Bool
isGridFilled grid = null [ () | Possible _ <- concat grid ]
isGridInvalid :: Grid -> Bool
isGridInvalid grid =
any isInvalidRow grid
|| any isInvalidRow (Data.List.transpose grid)
|| any isInvalidRow (subGridsToRows grid)
where
isInvalidRow row =
let fixeds = [x | Fixed x <- row]
emptyPossibles = [x | Possible x <- row, null x]
in hasDups fixeds || not (null emptyPossibles)
hasDups l = hasDups' l []
hasDups' [] _ = False
hasDups' (y:ys) xs
| y `elem` xs = True
| otherwise = hasDups' ys (y:xs)
isGridFilled
returns whether a grid is filled completely by checking it for any Possible
cells. isGridInvalid
checks if a grid is invalid because it either has duplicate fixed cells in any block or has any non-fixed cell with no possibilities.
Writing the solve
function is almost trivial now:
solve :: Grid -> Maybe Grid
solve grid = pruneGrid grid >>= solve'
where
solve' g
| isGridInvalid g = Nothing
| isGridFilled g = Just g
| otherwise =
let (grid1, grid2) = nextGrids g
in solve grid1 <|> solve grid2
We prune the grid as before and pipe it to the helper function solve'
. solve'
bails with a Nothing
if the grid is invalid, or returns the solved grid if it is filled completely. Otherwise, it finds the next two grids in the search tree and solves them recursively with backtracking by calling the solve
function. Backtracking here is implemented by the using the Alternative
(<|>
) implementation of the Maybe
type2. It takes the second branch in the computation if the first branch returns a Nothing
.
Whew! That took us long. Let’s put it to the final test now:
*Main> Just grid =
readGrid "6......1.4.........2...........5.4.7..8...3....1.9....3..4..2...5.1........8.6..."
*Main> putStr $ showGrid grid
6 . . . . . . 1 .
4 . . . . . . . .
. 2 . . . . . . .
. . . . 5 . 4 . 7
. . 8 . . . 3 . .
. . 1 . 9 . . . .
3 . . 4 . . 2 . .
. 5 . 1 . . . . .
. . . 8 . 6 . . .
*Main> Just grid' = solve grid
*Main> putStr $ showGrid grid'
6 9 3 7 8 4 5 1 2
4 8 7 5 1 2 9 3 6
1 2 5 9 6 3 8 7 4
9 3 2 6 5 1 4 8 7
5 6 8 2 4 7 3 9 1
7 4 1 3 9 8 6 2 5
3 1 9 4 7 5 2 6 8
8 5 6 1 2 9 7 4 3
2 7 4 8 3 6 1 5 9
It works! Let’s put a quick main
wrapper around solve
to call it from the command line:
main :: IO ()
main = do
inputs <- lines <$> getContents
Control.Monad.forM_ inputs $ \input ->
case readGrid input of
Nothing -> putStrLn "Invalid input"
Just grid -> case solve grid of
Nothing -> putStrLn "No solution found"
Just grid' -> putStrLn $ showGrid grid'
And now, we can invoke it from the command line:
$ echo ".......12.5.4............3.7..6..4....1..........8....92....8.....51.7.......3..." | stack exec sudoku
3 6 4 9 7 8 5 1 2
1 5 2 4 3 6 9 7 8
8 7 9 1 2 5 6 3 4
7 3 8 6 5 1 4 2 9
6 9 1 2 4 7 3 8 5
2 4 5 3 8 9 1 6 7
9 2 3 7 6 4 8 5 1
4 8 6 5 1 2 7 9 3
5 1 7 8 9 3 2 4 6
And, we are done.
If you want to play with different puzzles, the file here lists some of the toughest ones. Let’s run3 some of them through our program to see how fast it is:
$ head -n100 sudoku17.txt | time stack exec sudoku
... output omitted ...
116.70 real 198.09 user 94.46 sys
It took about 117 seconds to solve a hundred puzzles, so, about 1.2 seconds per puzzle. This is pretty slow but we’ll get around to making it faster in the subsequent posts.
In this rather verbose article, we learned how to write a simple Sudoku solver in Haskell step-by-step. In the later parts of this series, we’ll delve into profiling the solution and figuring out better algorithms and data structures to solve Sudoku more efficiently. The code till now is available here. Discuss this post on r/haskell or leave a comment.
If you liked this post, please leave a comment.
]]>You may find that context interesting.
The email which spawned this discussion was from a new employee, suggesting that we raise salaries. We discuss such organizational and infrastructural changes on occasion (not often) and an open conversation is welcome.
However, as I’m leaving nilenso in July, it felt worthwhile to capture what would otherwise be a one-time, long-winded email to nilenso’s existing staff and transform that into an article which could capture one founding member’s opinion of what nilenso is, for all future staff — and other fans of Employee-owned Co-operatives—who would care to read it.
The original email had a fairly simple 3-part structure.
First: How much can we pay? …without compromising on organizational goals like maintaining a rainy-day cash buffer and financing experimental new ventures?
Second, a declaration of our rates & salaries. Highlighted is the delta between the two. Our rates vary a lot from project to project, but for the sake of example, imagine our rates average to $100/hour. (100 is a nice round number.) At $100/hour, our rates were high for Indian software consultancies, but not astronomical. A napkin calculation roughs $100/hour to over $100,000/year in revenue. Historically, very senior people at nilenso made about half this revenue figure as salary.
[F]or people with more experience and established reputation, there is an incentive to break away and start their own business so that they can pocket the difference.
(All quotes are from the aforementioned email.)
Third and last, a spectrum of starting salaries. These were listed for college grads of reputable Indian universities, such as NIT and IIT. Paymasters near the top included Uber and Goldman Sachs. Near the bottom were Samsung and American Express.
Some of the people I know from college are very good programmers and I’ll have a very hard [time] convincing them to join Nilenso [sic], because the difference between our salaries and the industry “standard” is too large.
….
[M]ost of our clients can pay pretty high salaries. From their perspective it isn’t an unreasonable strategy to hire a bunch of folks from nilenso instead of negotiating our high rates
It is possible this argument doesn’t seem irrational in isolation, so let’s break it down part by part. From hereon out, please automatically prefix any declaration I make with “as of this writing.”
Let’s start from the end, since—tellingly—the actionable question was actually posed first.
There are a number of reasons our clients have not frequently hired our staff away. First, and most strictly, some contracts we sign have a two-way no-hire clause. In these cases, our clients cannot hire our employees and we cannot hire theirs. However, just because the contract does not forbid poaching does not make it a good idea. Our preferred engagement model is a long-term, cooperative and collaborative relationship; if either side starts yanking the other entity’s employees, that’s a great way to either poison the engagement.
In addition to all of this, however, there is sustainability at stake. As global consultancies go, nilenso is very small and very good. The author of the email is our 20th co-op member. Unless a client is very comfortable burning bridges, hiring two of our staff today trims our numbers by a non-trivial 10%. We don’t work with cannibal business owners.
That isn’t to say no one has ever left nilenso this way. If it happens, we simply hope that person takes their decision very seriously. Some of our alumni left to join past clients — and we wished them well.
People do not work at nilenso to get rich.
People work here because they believe in something. As we’ll see, “something” isn’t defined by some manifesto or mission statement. The “something” isn’t likely to be found elsewhere and it’s even less likely to be the same for two different people at nilenso. “Something” probably isn’t a single thing within the mind of a single person. “Something” is fractured and ever-changing. Each fractured shard is at any given time either a hallucination manifesting in action or a miscalculation fading to black.
As complex as The Somethings are, there has thus far been a few consistent threads. One? Technical excellence. A potential client from Vietnam recently asked “what is the ratio of fresh grads you hire to senior developers?” Though it may not appear so on the surface, the idea that “I’ll have a very hard [time] convincing [people I know from college] to join” stems from the very same misconception.
nilenso is not a leveraged consultancy.
The misconception is that nilenso desires to behave as other consultancies by “leveraging”** their weakest employees (read: fresh grads) and charging the highest possible rate by inflating them with as few “senior”** developers as possible. The misconception is not unfounded. Most consultancies operate this way. We do not.
** industry terminology, not mine
Convincing college grads to apply to nilenso is not a problem we’d like to solve. We have college grads begging us for internships and offering to work for free.
We do not want to hire college grads.
When our clients hire nilenso, we don’t want them satisfied. We want them elated. With one or two exceptions, our most junior developers always have multiple years of experience. They joined us because they couldn’t find their Something anywhere else in the industry. Not only has a college grad not, by definition, owned her own software in production — she hasn’t had an opportunity to go look for Something yet. Go searching first—and learn to build and deploy software while you’re at it. Maybe she’ll find her Something. If she doesn’t, she can join a company like ours.
We are not Uber or Goldman Sachs. We don’t treat our female colleagues like shit, we don’t have an “asshole culture”, we haven’t received a failing grade from the Better Business Bureau, nor do we have a list of “controversies and legal issues” as long as your arm.
Some of us have worked for companies like these and we have done everything in our power to prevent nilenso from becoming such a company. We don’t need revenue in the billions to behave this way but racing to compete salaries with such companies does us zero favours in that arena.
There is perhaps no better way for a college kid to learn the value of a company like nilenso than to experience a cesspool like Uber firsthand for a couple years. Not that I recommend the grads start their search there, mind you.
In “Part 3”, when I say that nilenso is “very small and very good” I mean that from the bottom of my heart. We are very good. We don’t fantasize about our clients paying us $100/hour — our current clients pay us this because we are worth it. Our first month with every new client is always evaluatory (we even did a 3-month evaluation period once). This is an opportunity for our clients to fire us if they find us too expensive and for us to fire them if we find the relationship isn’t productive.
Why are our developers worth well over $10,000 USD per month?
There is a watermark here. One can calculate the watermark in a number of ways. The first and most obvious is: How many people at nilenso have ever been paid $120,000/year or higher in any country? Two. So that probably puts the watermark too high. How many people at nilenso could be paid $120,000/year (roughly 77 lakh rupees) today — specifically in Bangalore? Three.
These numbers are clearly too low. The entire value of our company does not hinge on two or three people. But these are the only objective numbers we have on this axis. If we wanted to reach for a subjective number, we could say the watermark exists beneath anyone to whom our rates can be actively attributed. I would say these people number seven.
Seven of twenty active knowledge workers does not mean we “leverage” (ugh, that word) the other thirteen. Other folks are very close or arguably cross this watermark. But every team has a natural pecking order (a hierarchy drawn in a straight line, whether you draw it with a pen or with your mind) and wherever we assume this watermark to be, our value to our customers is more than the sum of the parts.
The flaw in the “wow, we make so much money!” argument is that no matter where you place this watermark, you must actively acknowledge from where this value is generated: above the watermark. If there is an argument to be made for altering the salary curve to reflect this, that alteration always implies an inflection point at the watermark. People above the watermark should be paid more. People below the watermark should be paid less. Your financial goals as an employee of such a company are dictated by your ability to cross the watermark and stay above it.
That doesn’t sound like a place I’d like to work.
I’m guessing the author of the original email feels the same way. Being our most junior staff member means starting at the very bottom of this curve and a search for the source of a consultancy’s value is always upward-facing. That’s a big hill to climb and it’s a lot more likely that crossing the watermark will involve nilenso growing from 20 to 40 rather than the most junior employee leapfrogging over 10 other people. We grow slowly on purpose — and it’s not to keep junior people at the bottom of the pay scale.
We would rather grow inside. We want our most junior staff to become more senior not by stacking themselves on top of new junior people but by perfecting their execution.
[F]or people with more experience and established reputation, there is an incentive to break away and start their own business so that they can pocket the difference.
No one at nilenso has ever done this. Why not?
Now we get to the juicy bits. The interesting question is not “how much can we pay?” at all. If someone wants to double her salary she can very easily do so today and nilenso is entirely unequipped to deal with that except to wish her the best with her new job.
People do not work at nilenso to get rich.
People work at nilenso to find… something. Something. What is that Something? To answer this, we must ask: What is nilenso? This is the interesting question.
Nilenso Software LLP is, today, a co-op; nilenso is self-hosted. It is written in a homoiconic language. It is self-referential. It is self-describing. The specification is the paper the specification is written on. And while none of these qualities are the Something we were all after, they are the qualities which equip us to do anything we want with the company. We are uninhibited in finding our Somethings if we never need to beg for VC money, never need to give up control, and never need to grow before we’re ready.
“Anything we want” can include bumping our salaries by 10% or 50%. It can include handing out bonuses. It can include democratically deciding we want to execute across-the-board pay cuts. But these are all pretty boring axes to manipulate the business. You don’t have to work at a co-op (and therefore run it) for very long to realize the the classic definitions of success are mind-numbingly boring… albeit always tempting.
Anything we want? Here is a far more interesting list:
Every rupee we keep inside nilenso, invest as nilenso, or spend as nilenso pushes us further toward a collective activity, a collective vision. Every rupee we divide and distribute, 5 paise per co-op member, is a step away from a collective vision. The farthest we can go in this direction is to distribute everything: nilenso saves nothing, invests nothing, and builds nothing. Well before that stage, nilenso becomes an Umbrella Consultancy and we are nilenso in name only… another army of contractors with a brand attached and a few rupees saved every month to pay the water bill.
It is entirely valid for the collective vision of nilenso to be “we don’t want to have a collective vision” but once that is decided, there is no going back. If you want to become a software product company (as we attempted with Relay and previously attempted with Kulu), an electric rickshaw taxi service (a business model we have seriously considered), or a cafe (a business model we have jokingly considered)… you can’t. You get to go back to Square One, where we were when we incorporated nilenso in 2013. But you get to go back with no money and no way to compete on salaries but to rebuild a vision from scratch.
Good luck. It wasn’t easy the first time.
There is no easy way to think about the future regarding corporate or personal finances and arrive at novel conclusions. Follow Steve Jobs’ advice, drop enough acid, and you may. But you are still unlikely to execute based on these conclusions when you are sober on Monday. Such contemplations demand an entire suite of thought exercises. This is my favourite:
Flip the Five Whys exercise around and you have the And Then What? exercise.
Once you are rich, then what? Once you have solved the original problem, then what? Once you have solved an entire category of problems, then what? Once you have taught others to do it, then what?
And then what?
And then what?
]]>When you hear the words “open source” attached to a product, what image comes to mind? Do you imagine an office full of designers obsessing over every little pixel? Or do you imagine an army of alpha-nerds piecing together a slap-dash knockoff of some proprietary software?
We all believe open source software is badly designed because, historically, it was. (#NotAllOpenSourceSoftware)
Thirty years ago, in 1988, the very concept of a license or design language for software was far beyond the average person. A decade later, everything was still awful. Software barely worked. Your PC definitely had a virus on it. HTML had a <blink> tag. Installing Linux required weeks of patience and belief in oneself — the kind of conviction I would need to train for a marathon. If there is one canonical failure we can promote as the poster child of this dark era, it is OpenOffice. Or StarOffice or LibreOffice or Apache OpenOffice or OOo or AOO or whatever name you might know it by. When the very name of the product is this broken it’s unlikely to succeed as a brand.
Fast forward ten more years. As of 2008, Firefox was eating Microsoft’s lunch. Firefox was open source, fast, and standards-compliant. It was easy to like Firefox. But we didn’t like Firefox. We loved Firefox. Firefox had a brand before it was Firefox. A brand isn’t a snappy name and a cute logo. Your product is your brand and your product is dictated by its design.
In 2018, user expectations demand that design asymptotically approach perfection: your product better be open, compatible, clean, fast, and powerful. When nilenso first made the move from Slack to Mattermost, we kept all of this in mind. From our very first deployment of Mattermost it felt whole. We chose Mattermost because it was so very close to the design we wanted, we knew we could help bring it the rest of the way.
The design problems that currently exist with Mattermost aren’t just visual but interaction and experience related. One of the first issues we took up was adding infinite scroll to the channel history. We couldn’t fathom why users should manually click to load every page of history. So we fixed it!
In the not-too-distant future we will add all the little nuanced details which transform Relay into software you absolutely love. At the moment, however, our aim is to solve the obvious problems our users have with the Mattermost/Relay experience. Like infinite scroll, these are the problems that left our early users shouting at us “ugh… how have you not fixed this yet?!”
Such problems stem from the pains caused by inconsistencies and visual discrepancies which hamper usability. It is surprising to many how quickly these seemingly small issues amount to death by a thousand papercuts. It is equally surprising how quickly software becomes a joy to use once you get the basics right.
We’ll start with the basics — icons and fonts — and then move on to the specifics of how we redesigned Mattermost as Relay, piece by piece.
When we moved from Slack to Mattermost the difference in icons used was stark. All icons used in Mattermost were inconsistent with respect to weights. They were standing out of the interface rather than blending in.
After further digging, we realised that Mattermost uses the free version of Font Awesome. Don’t get me wrong, I love the guys at Font Awesome; I even backed their kickstarter. But not all icons being available in the free weights (regular & solid) is quite a bummer.
We had three rules for selecting an icon set:
Allow me to interrupt with an aside. Technically, it is possible for us to build commercial artefacts into the Relay client by keeping them out of the source tree, even though Relay is AGPL3-licensed (as a derivative of the mattermost-webapp). Why, then, do we feel it is essential that the icons be open source? Relay lives at the intersection of the principles of Free Software and the license which governs it. Relay stands on the shoulders of giants — not only the Mattermost source code but Linux, GoLang, React, and hundreds of other projects. Here, between the spirit and the letter of Open Philosophies, exists a great deal of nuance. Cutting through this nuance is a root concept, however: “share alike.” If we take something from the free/open software community, what we return to that community should afford the next recipient the same freedoms we had. If we tweak Relay’s design until it is absolutely perfect but that involves changes which Mattermost cannot absorb upstream, we have failed them and every other project we have built ourselves upon. Thus, our icons must have an open license.
— Steven Deobald
Thanks Steve. Keeping the above mentioned points in mind, there were three icon sets that made the final round of discussion:
We chose the Clarity icon set because of styles, consistency, and the icons being part of a design language system. Clarity is an open source design system created and maintained by the good folks at VMware. Since VMware has a dedicated team pushing continuous updates there is little fear the icon set will be abandoned or go stale.
Our font selection followed the same rules:
…along with these additions:
We stress so much on font weights because Relay is, first and foremost, a messaging tool. In Relay, you spend most of your time reading and we want this experience to be as enjoyable as possible. Relay also has complete Markdown support and weights help establish hierarchy both within the app and within the messages themselves.
We had a few contenders which satisfied these requirements. PT Sans, Lato, and Noto Sans all have an enjoyable reading experience but after a lot of trial and error we chose Fira Sans.
Fira Sans is an incredible font. It has a healthy variety of weights and the compressed nature of the font holds the messaging interface together while keeping it crisp.
Did you know that Fira Sans was built on top of Erik Spiekermann’s commercial font FF Meta? And that Spiekermann was commissioned by the Mozilla Foundation! Read the whole story here
Implementing the design basics improved the existing experience. But to make those changes truly delightful we need to fine-tune every aspect of the app. We have started with three major sections:
The sidebar’s Team section allows users to switch between teams within organisations that they’re a part of. The current Mattermost implementation truncates the team name and adds a byline. The problem with this is that it does not add any value since hovering over the team opens a tooltip with the team name anyway. We removed the byline and the truncated text.
The sidebar’s Channel section felt mashed together. There is a clear hierarchy intended, which includes channel headers such as “Favorite Channels” and a distinction between active and inactive channels. We brought this hierarchy to the surface. Active channels have a heavy weight and inactive channels fade away with a lower opacity, despite the increased crispness of Fira Sans.
Grouped channels now have tighter spacing and groups themselves have larger spacing, allowing group titles to have a smaller font. All together, this makes the hierarchy obvious even at a glance.
In the channel header we cleaned up the icons. Here are the issues we encountered with the icons:
The old icons for the channel header stood out, drawing attention away from the main chat area. We wanted them instead to blend into the design — unobtrusive but still obvious. To do this we removed the borders and switched the icon set to Clarity.
In addition to icons, we fixed the CSS for the header and right sidebar, getting rid of all funny margins and relative positioning. We also added consistent heights, paddings, and sizes for each of the icons. These fixes don’t just add polish to the UI; once merged, they will also ensure it is easier for anyone in the community, including us, to customise Mattermost.
The messaging interface is where users spend the most time. The messaging interface is also where Mattermost has most of its design inconsistencies, which are most glaring in the basics: alignment, spacing between posts, font size, and other elements on the interface such as the reply box.
After we moved to Fira Sans we bumped up the font size, made sure the headings and names are bolded to establish hierarchy, and fixed alignment and spacing issues to deliver a crisper and more enjoyable reading experience.
Accessibility matters. We have light-sensitive coworkers and we’re sure you do, too. That’s why Relay comes with light and dark themes out of the box.
There are many Relay/Mattermost-compatible themes available already but we will pour all our energy into these two. They already make Relay beautiful and with every kaizen they will only get better.
Of course, we don’t restrict you to these two themes. Relay also comes with community themes and Relay’s entire colour scheme is fully customisable… not just the sidebar (*cough* Slack *cough*). We are working to make Relay 100% customisable through CSS so users have complete control over every aspect of the UI, if they want.
By following a consistent design language for icons, font, and themes we ensure all our “pieces” are whole unto themselves. Putting them back together, the application itself becomes whole.
Mattermost has published a design challenge that encourages designers from all over the world to improve Mattermost (and, hence, Relay) readability and usability. Our redesign addresses this challenge.
Sign up, try the redesign, and tell us what you think!
Along with visual design changes, Relay also aims to significantly improve its user experience. Our public Trello board contains our plan for Relay’s future.
“Design” is the 8th most popular tag on Medium. “Open Source” is ranked 290. Source code licenses will never be sexy for the same reason your city’s water utility company will never be. The water supply is inherently a part of your city’s design. Similarly, your freedom as a user is designed into software you buy. Your ownership of a product is but one more limb in the tree of design.
Open source software needs good design because good design respects you, the user.
❤
Designing for Open Source Software was originally published in The Relay on Medium, where people are continuing the conversation by highlighting and responding to this story.
]]>But first,
What happens if we hit a URL on our server which does not exist? Let’s fire up the server and test it:
$ http GET https://localhost:4000/v1/random
HTTP/1.1 404 Not Found
Connection: keep-alive
Content-Length: 148
Content-Security-Policy: default-src 'self'
Content-Type: text/html; charset=utf-8
Date: Sat, 30 Sep 2017 08:23:20 GMT
X-Content-Type-Options: nosniff
X-Powered-By: Express
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /v1/random</pre>
</body>
</html>
We get back a default HTML response with a 404 status from Express. Since we are writing a JSON API, we should return a JSON response in this case too. We add the following code in the src/SimpleService/Server.purs
file to add a catch-all route and send a 404 status with a JSON error message:
-- previous code
import Data.Either (fromRight)
import Data.String.Regex (Regex, regex) as Re
import Data.String.Regex.Flags (noFlags) as Re
import Node.Express.App (App, all, delete, get, http, listenHttp, post, useExternal)
import Node.Express.Response (sendJson, setStatus)
import Partial.Unsafe (unsafePartial)
-- previous code
allRoutePattern :: Re.Regex
allRoutePattern = unsafePartial $ fromRight $ Re.regex "/.*" Re.noFlags
app :: forall eff. PG.Pool -> App (postgreSQL :: PG.POSTGRESQL | eff)
app pool = do
useExternal jsonBodyParser
get "/v1/user/:id" $ getUser pool
delete "/v1/user/:id" $ deleteUser pool
post "/v1/users" $ createUser pool
patch "/v1/user/:id" $ updateUser pool
get "/v1/users" $ listUsers pool
all allRoutePattern do
setStatus 404
sendJson {error: "Route not found"}
where
patch = http (CustomMethod "patch")
allRoutePattern
matches all routes because it uses a "/.*"
regular expression. We place it as the last route to match all the otherwise unrouted requests. Let’s see what is the result:
$ http GET https://localhost:4000/v1/random
HTTP/1.1 404 Not Found
Connection: keep-alive
Content-Length: 27
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 08:46:46 GMT
ETag: W/"1b-772e0u4nrE48ogbR0KmKfSvrHUE"
X-Powered-By: Express
{
"error": "Route not found"
}
Now we get a nicely formatted JSON response.
Another scenario is when our application throws some uncaught error. To simulate this, we shut down our postgres database and hit the server for listing users:
$ http GET https://localhost:4000/v1/users
HTTP/1.1 500 Internal Server Error
Connection: keep-alive
Content-Length: 372
Content-Security-Policy: default-src 'self'
Content-Type: text/html; charset=utf-8
Date: Sat, 30 Sep 2017 08:53:40 GMT
X-Content-Type-Options: nosniff
X-Powered-By: Express
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Error: connect ECONNREFUSED 127.0.0.1:5432<br> at Object._errnoException (util.js:1026:11)<br> at _exceptionWithHostPort (util.js:1049:20)<br> at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)</pre>
</body>
</html>
We get another default HTML response from Express with a 500 status. Again, in this case we’d like to return a JSON response. We add the following code to the src/SimpleService/Server.purs
file:
-- previous code
import Control.Monad.Eff.Exception (message)
import Node.Express.App (App, all, delete, get, http, listenHttp, post, useExternal, useOnError)
-- previous code
app :: forall eff. PG.Pool -> App (postgreSQL :: PG.POSTGRESQL | eff)
app pool = do
-- previous code
useOnError \err -> do
setStatus 500
sendJson {error: message err}
where
patch = http (CustomMethod "patch")
We add the useOnError
handler which comes with purescript-express
to return the error message as a JSON response. Back on the command-line:
$ http GET https://localhost:4000/v1/users
HTTP/1.1 500 Internal Server Error
Connection: keep-alive
Content-Length: 47
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 09:01:37 GMT
ETag: W/"2f-cJuIW6961YCpo9TWDSZ9VWHLGHE"
X-Powered-By: Express
{
"error": "connect ECONNREFUSED 127.0.0.1:5432"
}
It works! Bugs are fixed now. We proceed to add next features.
Let’s recall the code to update a user from the src/SimpleService/Handler.purs
file:
updateUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
updateUser pool = getRouteParam "id" >>= case _ of
Nothing -> respond 422 { error: "User ID is required" }
Just sUserId -> case fromString sUserId of
Nothing -> respond 422 { error: "User ID must be positive: " <> sUserId }
Just userId -> getBody >>= case _ of
Left errs -> respond 422 { error: intercalate ", " $ map renderForeignError errs}
Right (UserPatch userPatch) -> case unNullOrUndefined userPatch.name of
Nothing -> respondNoContent 204
Just userName -> if userName == ""
then respond 422 { error: "User name must not be empty" }
else do
savedUser <- liftAff $ PG.withConnection pool \conn -> PG.withTransaction conn do
P.findUser conn userId >>= case _ of
Nothing -> pure Nothing
Just (User user) -> do
let user' = User (user { name = userName })
P.updateUser conn user'
pure $ Just user'
case savedUser of
Nothing -> respond 404 { error: "User not found with id: " <> sUserId }
Just user -> respond 200 (encode user)
As we can see, the actual request handling logic is obfuscated by the request validation logic for the user id and the user name patch parameters. We also notice that we are using three constructs for validation here: Maybe
, Either
and if-then-else
. However, we can use just Either
to subsume all these cases as it can “carry” a failure as well as a success case. Either
also comes with a nice monad transformer ExceptT
which provides the do
syntax for failure propagation. So we choose ExceptT
as the base construct for our validation framework and write functions to upgrade Maybe
and if-then-else
to it. We add the following code to the src/SimpleService/Validation.purs
file:
module SimpleService.Validation
(module MoreExports, module SimpleService.Validation) where
import Prelude
import Control.Monad.Except (ExceptT, except, runExceptT)
import Data.Either (Either(..))
import Data.Maybe (Maybe(..))
import Node.Express.Handler (HandlerM, Handler)
import Node.Express.Response (sendJson, setStatus)
import Node.Express.Types (EXPRESS)
import Control.Monad.Except (except) as MoreExports
type Validation eff a = ExceptT String (HandlerM (express :: EXPRESS | eff)) a
exceptMaybe :: forall e m a. Applicative m => e -> Maybe a -> ExceptT e m a
exceptMaybe e a = except $ case a of
Just x -> Right x
Nothing -> Left e
exceptCond :: forall e m a. Applicative m => e -> (a -> Boolean) -> a -> ExceptT e m a
exceptCond e cond a = except $ if cond a then Right a else Left e
withValidation :: forall eff a. Validation eff a -> (a -> Handler eff) -> Handler eff
withValidation action handler = runExceptT action >>= case _ of
Left err -> do
setStatus 422
sendJson {error: err}
Right x -> handler x
We re-export except
from the Control.Monad.Except
module. We also add a withValidation
function which runs an ExceptT
based validation and either returns an error response with a 422 status in case of a failed validation or runs the given action with the valid value in case of a successful validation.
Using these functions, we now write updateUser
in the src/SimpleService/Handler.purs
file as:
-- previous code
import Control.Monad.Trans.Class (lift)
import Data.Bifunctor (lmap)
import Data.Foreign (ForeignError, renderForeignError)
import Data.List.NonEmpty (toList)
import Data.List.Types (NonEmptyList)
import Data.Tuple (Tuple(..))
import SimpleService.Validation as V
-- previous code
renderForeignErrors :: forall a. Either (NonEmptyList ForeignError) a -> Either String a
renderForeignErrors = lmap (toList >>> map renderForeignError >>> intercalate ", ")
updateUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
updateUser pool = V.withValidation (Tuple <$> getUserId <*> getUserPatch)
\(Tuple userId (UserPatch userPatch)) ->
case unNullOrUndefined userPatch.name of
Nothing -> respondNoContent 204
Just uName -> V.withValidation (getUserName uName) \userName -> do
savedUser <- liftAff $ PG.withConnection pool \conn -> PG.withTransaction conn do
P.findUser conn userId >>= case _ of
Nothing -> pure Nothing
Just (User user) -> do
let user' = User (user { name = userName })
P.updateUser conn user'
pure $ Just user'
case savedUser of
Nothing -> respond 404 { error: "User not found with id: " <> show userId }
Just user -> respond 200 (encode user)
where
getUserId = lift (getRouteParam "id")
>>= V.exceptMaybe "User ID is required"
>>= fromString >>> V.exceptMaybe "User ID must be positive"
getUserPatch = lift getBody >>= V.except <<< renderForeignErrors
getUserName = V.exceptCond "User name must not be empty" (_ == "")
The validation logic has been extracted out in separate functions now which are composed using Applicative. The validation steps are composed using the ExceptT
monad. We are now free to express the core logic of the function clearly. We rewrite the src/SimpleService/Handler.purs
file using the validations:
module SimpleService.Handler where
import Prelude
import Control.Monad.Aff.Class (liftAff)
import Control.Monad.Trans.Class (lift)
import Data.Bifunctor (lmap)
import Data.Either (Either)
import Data.Foldable (intercalate)
import Data.Foreign (ForeignError, renderForeignError)
import Data.Foreign.Class (encode)
import Data.Foreign.NullOrUndefined (unNullOrUndefined)
import Data.Int (fromString)
import Data.List.NonEmpty (toList)
import Data.List.Types (NonEmptyList)
import Data.Maybe (Maybe(..))
import Data.Tuple (Tuple(..))
import Database.PostgreSQL as PG
import Node.Express.Handler (Handler)
import Node.Express.Request (getBody, getRouteParam)
import Node.Express.Response (end, sendJson, setStatus)
import SimpleService.Persistence as P
import SimpleService.Validation as V
import SimpleService.Types
getUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
getUser pool = V.withValidation getUserId \userId ->
liftAff (PG.withConnection pool $ flip P.findUser userId) >>= case _ of
Nothing -> respond 404 { error: "User not found with id: " <> show userId }
Just user -> respond 200 (encode user)
deleteUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
deleteUser pool = V.withValidation getUserId \userId -> do
found <- liftAff $ PG.withConnection pool \conn -> PG.withTransaction conn do
P.findUser conn userId >>= case _ of
Nothing -> pure false
Just _ -> do
P.deleteUser conn userId
pure true
if found
then respondNoContent 204
else respond 404 { error: "User not found with id: " <> show userId }
createUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
createUser pool = V.withValidation getUser \user@(User _) -> do
liftAff (PG.withConnection pool $ flip P.insertUser user)
respondNoContent 201
where
getUser = lift getBody
>>= V.except <<< renderForeignErrors
>>= V.exceptCond "User ID must be positive" (\(User user) -> user.id > 0)
>>= V.exceptCond "User name must not be empty" (\(User user) -> user.name /= "")
updateUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
updateUser pool = V.withValidation (Tuple <$> getUserId <*> getUserPatch)
\(Tuple userId (UserPatch userPatch)) ->
case unNullOrUndefined userPatch.name of
Nothing -> respondNoContent 204
Just uName -> V.withValidation (getUserName uName) \userName -> do
savedUser <- liftAff $ PG.withConnection pool \conn -> PG.withTransaction conn do
P.findUser conn userId >>= case _ of
Nothing -> pure Nothing
Just (User user) -> do
let user' = User (user { name = userName })
P.updateUser conn user'
pure $ Just user'
case savedUser of
Nothing -> respond 404 { error: "User not found with id: " <> show userId }
Just user -> respond 200 (encode user)
where
getUserPatch = lift getBody >>= V.except <<< renderForeignErrors
getUserName = V.exceptCond "User name must not be empty" (_ /= "")
listUsers :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
listUsers pool = liftAff (PG.withConnection pool P.listUsers) >>= encode >>> respond 200
getUserId :: forall eff. V.Validation eff Int
getUserId = lift (getRouteParam "id")
>>= V.exceptMaybe "User ID is required"
>>= fromString >>> V.exceptMaybe "User ID must be an integer"
>>= V.exceptCond "User ID must be positive" (_ > 0)
renderForeignErrors :: forall a. Either (NonEmptyList ForeignError) a -> Either String a
renderForeignErrors = lmap (toList >>> map renderForeignError >>> intercalate ", ")
respond :: forall eff a. Int -> a -> Handler eff
respond status body = do
setStatus status
sendJson body
respondNoContent :: forall eff. Int -> Handler eff
respondNoContent status = do
setStatus status
end
The code is much cleaner now. Let’s try out a few test cases:
$ http POST https://localhost:4000/v1/users id:=3 name=roger
HTTP/1.1 201 Created
Connection: keep-alive
Content-Length: 0
Date: Sat, 30 Sep 2017 12:13:37 GMT
X-Powered-By: Express
$ http POST https://localhost:4000/v1/users id:=3
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 102
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:13:50 GMT
ETag: W/"66-/c4cfoquQZGwtDBUzHjJydJAHJ0"
X-Powered-By: Express
{
"error": "Error at array index 0: (ErrorAtProperty \"name\" (TypeMismatch \"String\" \"Undefined\"))"
}
$ http POST https://localhost:4000/v1/users id:=3 name=""
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 39
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:14:02 GMT
ETag: W/"27-JQsh12xu/rEFdWy8REF4NMtBUB4"
X-Powered-By: Express
{
"error": "User name must not be empty"
}
$ http POST https://localhost:4000/v1/users id:=0 name=roger
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 36
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:14:14 GMT
ETag: W/"24-Pvt1L4eGilBmVtaOGHlSReJ413E"
X-Powered-By: Express
{
"error": "User ID must be positive"
}
$ http GET https://localhost:4000/v1/user/3
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 23
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:14:28 GMT
ETag: W/"17-1scpiB1FT9DBu9s4I1gNWSjH2go"
X-Powered-By: Express
{
"id": 3,
"name": "roger"
}
$ http GET https://localhost:4000/v1/user/asdf
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 38
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:14:40 GMT
ETag: W/"26-//tvORl1gGDUMwgSaqbEpJhuadI"
X-Powered-By: Express
{
"error": "User ID must be an integer"
}
$ http GET https://localhost:4000/v1/user/-1
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 36
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:14:45 GMT
ETag: W/"24-Pvt1L4eGilBmVtaOGHlSReJ413E"
X-Powered-By: Express
{
"error": "User ID must be positive"
}
It works as expected.
Right now our application configuration resides in the main
function:
main = runServer port databaseConfig
where
port = 4000
databaseConfig = { user: "abhinav"
, password: ""
, host: "localhost"
, port: 5432
, database: "simple_service"
, max: 10
, idleTimeoutMillis: 1000
}
We are going to extract it out of the code and read it from the environment variables using the purescript-config
package. First, we install the required packages using bower.
Now, we write the following code in the src/SimpleService/Config.purs
file:
module SimpleService.Config where
import Data.Config
import Prelude
import Control.Monad.Eff (Eff)
import Data.Config.Node (fromEnv)
import Data.Either (Either)
import Data.Set (Set)
import Database.PostgreSQL as PG
import Node.Process (PROCESS)
type ServerConfig =
{ port :: Int
, databaseConfig :: PG.PoolConfiguration
}
databaseConfig :: Config {name :: String} PG.PoolConfiguration
databaseConfig =
{ user: _, password: _, host: _, port: _, database: _, max: _, idleTimeoutMillis: _ }
<$> string {name: "user"}
<*> string {name: "password"}
<*> string {name: "host"}
<*> int {name: "port"}
<*> string {name: "database"}
<*> int {name: "pool_size"}
<*> int {name: "idle_conn_timeout_millis"}
portConfig :: Config {name :: String} Int
portConfig = int {name: "port"}
serverConfig :: Config {name :: String} ServerConfig
serverConfig =
{ port: _, databaseConfig: _}
<$> portConfig
<*> prefix {name: "db"} databaseConfig
readServerConfig :: forall eff.
Eff (process :: PROCESS | eff) (Either (Set String) ServerConfig)
readServerConfig = fromEnv "SS" serverConfig
We use the applicative DSL provided in Data.Config
module to build a description of our configuration. This description contains the keys and types of the configuration, for consumption by various interpreters. Then we use the fromEnv
interpreter to read the config from the environment variables derived from the name
fields in the records in the description in the readServerConfig
function. We also write a bash script to set those environment variables in the development environment in the setenv.sh
file:
export SS_PORT=4000
export SS_DB_USER="abhinav"
export SS_DB_PASSWORD=""
export SS_DB_HOST="localhost"
export SS_DB_PORT=5432
export SS_DB_DATABASE="simple_service"
export SS_DB_POOL_SIZE=10
export SS_DB_IDLE_CONN_TIMEOUT_MILLIS=1000
Now we rewrite our src/Main.purs
file to use the readServerConfig
function:
module Main where
import Prelude
import Control.Monad.Eff (Eff)
import Control.Monad.Eff.Console (CONSOLE, log)
import Data.Either (Either(..))
import Data.Set (toUnfoldable)
import Data.String (joinWith)
import Database.PostgreSQL as PG
import Node.Express.Types (EXPRESS)
import Node.Process (PROCESS)
import Node.Process as Process
import SimpleService.Config (readServerConfig)
import SimpleService.Server (runServer)
main :: forall eff. Eff ( console :: CONSOLE
, express :: EXPRESS
, postgreSQL :: PG.POSTGRESQL
, process :: PROCESS
| eff ) Unit
main = readServerConfig >>= case _ of
Left missingKeys -> do
log $ "Unable to start. Missing Env keys: " <> joinWith ", " (toUnfoldable missingKeys)
Process.exit 1
Right { port, databaseConfig } -> runServer port databaseConfig
If readServerConfig
fails, we print the missing keys to the console and exit the process. Else we run the server with the read config.
To test this, we stop the server we ran in the beginning, source the config, and run it again:
$ pulp --watch run
* Building project in /Users/abhinav/ps-simple-rest-service
* Build successful.
Server listening on :4000
^C
$ source setenv.sh
$ pulp --watch run
* Building project in /Users/abhinav/ps-simple-rest-service
* Build successful.
Server listening on :4000
It works! We test the failure case by opening another terminal which does not have the environment variables set:
$ pulp run
* Building project in /Users/abhinav/ps-simple-rest-service
* Build successful.
Unable to start. Missing Env keys: SS_DB_DATABASE, SS_DB_HOST, SS_DB_IDLE_CONN_TIMEOUT_MILLIS, SS_DB_PASSWORD, SS_DB_POOL_SIZE, SS_DB_PORT, SS_DB_USER, SS_PORT
* ERROR: Subcommand terminated with exit code 1
Up next, we add logging to our application.
For logging, we use the purescript-logging
package. We write a logger which logs to stdout
; in the src/SimpleService/Logger.purs
file:
module SimpleService.Logger
( debug
, info
, warn
, error
) where
import Prelude
import Control.Logger as L
import Control.Monad.Eff.Class (class MonadEff, liftEff)
import Control.Monad.Eff.Console as C
import Control.Monad.Eff.Now (NOW, now)
import Data.DateTime.Instant (toDateTime)
import Data.Either (fromRight)
import Data.Formatter.DateTime (Formatter, format, parseFormatString)
import Data.Generic.Rep (class Generic)
import Data.Generic.Rep.Show (genericShow)
import Data.String (toUpper)
import Partial.Unsafe (unsafePartial)
data Level = Debug | Info | Warn | Error
derive instance eqLevel :: Eq Level
derive instance ordLevel :: Ord Level
derive instance genericLevel :: Generic Level _
instance showLevel :: Show Level where
show = toUpper <<< genericShow
type Entry =
{ level :: Level
, message :: String
}
dtFormatter :: Formatter
dtFormatter = unsafePartial $ fromRight $ parseFormatString "YYYY-MM-DD HH:mm:ss.SSS"
logger :: forall m e. (
MonadEff (console :: C.CONSOLE, now :: NOW | e) m) => L.Logger m Entry
logger = L.Logger $ \{ level, message } -> liftEff do
time <- toDateTime <$> now
C.log $ "[" <> format dtFormatter time <> "] " <> show level <> " " <> message
log :: forall m e.
MonadEff (console :: C.CONSOLE , now :: NOW | e) m
=> Entry -> m Unit
log entry@{level} = L.log (L.cfilter (\e -> e.level == level) logger) entry
debug :: forall m e.
MonadEff (console :: C.CONSOLE , now :: NOW | e) m => String -> m Unit
debug message = log { level: Debug, message }
info :: forall m e.
MonadEff (console :: C.CONSOLE , now :: NOW | e) m => String -> m Unit
info message = log { level: Info, message }
warn :: forall m e.
MonadEff (console :: C.CONSOLE , now :: NOW | e) m => String -> m Unit
warn message = log { level: Warn, message }
error :: forall m e.
MonadEff (console :: C.CONSOLE , now :: NOW | e) m => String -> m Unit
error message = log { level: Error, message }
purescript-logging
lets us define our own logging levels and loggers. We define four log levels, and a log entry type with the log level and the message. Then we write the logger which will print the log entry to stdout
along with the current time as a well formatted string. We define convenience functions for each log level.
Before we proceed, let’s install the required dependencies.
Now we add a request logger middleware to our server in the src/SimpleService/Server.purs
file:
-- previous code
import Control.Monad.Eff.Console (CONSOLE)
import Control.Monad.Eff.Now (NOW)
import Data.Maybe (maybe)
import Data.String (toUpper)
import Node.Express.App (App, all, delete, get, http, listenHttp, post, use, useExternal, useOnError)
import Node.Express.Handler (Handler, next)
import Node.Express.Request (getMethod, getPath)
import SimpleService.Logger as Log
-- previous code
requestLogger :: forall eff. Handler (console :: CONSOLE, now :: NOW | eff)
requestLogger = do
method <- getMethod
path <- getPath
Log.debug $ "HTTP: " <> maybe "" id ((toUpper <<< show) <$> method) <> " " <> path
next
app :: forall eff.
PG.Pool
-> App (postgreSQL :: PG.POSTGRESQL, console :: CONSOLE, now :: NOW | eff)
app pool = do
useExternal jsonBodyParser
use requestLogger
-- previous code
We also convert all our previous logging statements which used Console.log
to use SimpleService.Logger
and add logs in our handlers. We can see logging in effect by restarting the server and hitting it:
$ pulp --watch run
* Building project in /Users/abhinav/ps-simple-rest-service
* Build successful.
[2017-09-30 16:02:41.634] INFO Server listening on :4000
[2017-09-30 16:02:43.494] DEBUG HTTP: PATCH /v1/user/3
[2017-09-30 16:02:43.517] DEBUG Updated user: 3
[2017-09-30 16:03:46.615] DEBUG HTTP: DELETE /v1/user/3
[2017-09-30 16:03:46.635] DEBUG Deleted user 3
[2017-09-30 16:05:03.805] DEBUG HTTP: GET /v1/users
In this tutorial we learned how to create a simple JSON REST web service written in PureScript with persistence, validation, configuration and logging. The complete code for this tutorial can be found in github. Discuss this post in the comments.
If you liked this post, please leave a comment.
]]>The aim of this two-part tutorial is to create a simple JSON REST web service written in PureScript, to run on a node.js server. This assumes that you have basic proficiency with PureScript. We have the following requirements:
In this part we’ll work on setting up the project and on the first two requirements. In the next part we’ll work on the rest of the requirements.
We start with installing PureScript and the required tools. This assumes that we have node and npm installed on our machine.
Pulp is a build tool for PureScript projects and bower is a package manager used to get PureScript libraries. We’ll have to add ~/.local/bin
in our $PATH
(if it is not already added) to access the binaries installed.
Let’s create a directory for our project and make Pulp initialize it:
$ mkdir ps-simple-rest-service
$ cd ps-simple-rest-service
$ pulp init
$ ls
bower.json bower_components src test
$ cat bower.json
{
"name": "ps-simple-rest-service",
"ignore": [
"**/.*",
"node_modules",
"bower_components",
"output"
],
"dependencies": {
"purescript-prelude": "^3.1.0",
"purescript-console": "^3.0.0"
},
"devDependencies": {
"purescript-psci-support": "^3.0.0"
}
}
$ ls bower_components
purescript-console purescript-eff purescript-prelude purescript-psci-support
Pulp creates the basic project structure for us. src
directory will contain the source while the test
directory will contain the tests. bower.json
contains the PureScript libraries as dependencies which are downloaded and installed in the bower_components
directory.
First, we create the types needed in src/SimpleService/Types.purs
:
module SimpleService.Types where
import Prelude
import Data.Foreign.Class (class Decode, class Encode)
import Data.Foreign.Generic (defaultOptions, genericDecode, genericEncode)
import Data.Generic.Rep (class Generic)
import Data.Generic.Rep.Show (genericShow)
type UserID = Int
newtype User = User
{ id :: UserID
, name :: String
}
derive instance genericUser :: Generic User _
instance showUser :: Show User where
show = genericShow
instance decodeUser :: Decode User where
decode = genericDecode $ defaultOptions { unwrapSingleConstructors = true }
instance encodeUser :: Encode User where
encode = genericEncode $ defaultOptions { unwrapSingleConstructors = true }
We are using the generic support for PureScript types from the purescript-generics-rep
and purescript-foreign-generic
libraries to encode and decode the User
type to JSON. We install the library by running the following command:
Now we can load up the module in the PureScript REPL and try out the JSON conversion features:
$ pulp repl
> import SimpleService.Types
> user = User { id: 1, name: "Abhinav"}
> user
(User { id: 1, name: "Abhinav" })
> import Data.Foreign.Generic
> userJSON = encodeJSON user
> userJSON
"{\"name\":\"Abhinav\",\"id\":1}"
> import Data.Foreign
> import Control.Monad.Except.Trans
> import Data.Identity
> dUser = decodeJSON userJSON :: F User
> eUser = let (Identity eUser) = runExceptT $ dUser in eUser
> eUser
(Right (User { id: 1, name: "Abhinav" }))
We use encodeJSON
and decodeJSON
functions from the Data.Foreign.Generic
module to encode and decode the User
instance to JSON. The return type of decodeJSON
is a bit complicated as it needs to return the parsing errors too. In this case, the decoding returns no errors and we get back a Right
with the correctly parsed User
instance.
Next, we add the support for saving a User
instance to a Postgres database. First, we install the required libraries using bower and npm: pg
for Javascript bindings to call Postgres, purescript-aff
for asynchronous processing and purescript-postgresql-client
for PureScript wrapper over pg
:
$ npm init -y
$ npm install pg@6.4.0 --save
$ bower install purescript-aff --save
$ bower install purescript-postgresql-client --save
Before writing the code, we create the database and the users
table using the command-line Postgres client:
$ psql postgres
psql (9.5.4)
Type "help" for help.
postgres=# create database simple_service;
CREATE DATABASE
postgres=# \c simple_service
You are now connected to database "simple_service" as user "abhinav".
simple_service=# create table users (id int primary key, name varchar(100) not null);
CREATE TABLE
simple_service=# \d users
Table "public.users"
Column | Type | Modifiers
--------+------------------------+-----------
id | integer | not null
name | character varying(100) | not null
Indexes:
"users_pkey" PRIMARY KEY, btree (id)
Now we add support for converting a User
instance to-and-from an SQL row by adding the following code in the src/SimpleService/Types.purs
file:
import Data.Array as Array
import Data.Either (Either(..))
import Database.PostgreSQL (class FromSQLRow, class ToSQLRow, fromSQLValue, toSQLValue)
-- code written earlier
instance userFromSQLRow :: FromSQLRow User where
fromSQLRow [id, name] =
User <$> ({ id: _, name: _} <$> fromSQLValue id <*> fromSQLValue name)
fromSQLRow xs = Left $ "Row has " <> show n <> " fields, expecting 2."
where n = Array.length xs
instance userToSQLRow :: ToSQLRow User where
toSQLRow (User {id, name}) = [toSQLValue id, toSQLValue name]
We can try out the persistence support in the REPL:
$ pulp repl
PSCi, version 0.11.6
Type :? for help
import Prelude
>
> import SimpleService.Types
> import Control.Monad.Aff (launchAff, liftEff')
> import Database.PostgreSQL as PG
> user = User { id: 1, name: "Abhinav" }
> databaseConfig = {user: "abhinav", password: "", host: "localhost", port: 5432, database: "simple_service", max: 10, idleTimeoutMillis: 1000}
> :paste
… void $ launchAff do
… pool <- PG.newPool databaseConfig
… PG.withConnection pool $ \conn -> do
… PG.execute conn (PG.Query "insert into users (id, name) values ($1, $2)") user
…
unit
> import Data.Foldable (for_)
> import Control.Monad.Eff.Console (logShow)
> :paste
… void $ launchAff do
… pool <- PG.newPool databaseConfig
… PG.withConnection pool $ \conn -> do
… users :: Array User <- PG.query conn (PG.Query "select id, name from users where id = $1") (PG.Row1 1)
… liftEff' $ void $ for_ users logShow
…
unit
(User { id: 1, name: "Abhinav" })
We create the databaseConfig
record with the configs needed to connect to the database. Using the record, we create a new Postgres connection pool (PG.newPool
) and get a connection from it (PG.withConnection
). We call PG.execute
with the connection, the SQL insert query for the users table and the User
instance, to insert the user into the table. All of this is done inside launchAff
which takes care of sequencing the callbacks correctly to make the asynchronous code look synchronous.
Similarly, in the second part, we query the table using PG.query
function by calling it with a connection, the SQL select query and the User
ID as the query parameter. It returns an Array
of users which we log to the console using the logShow
function.
We use this experiment to write the persistence related code in the src/SimpleService/Persistence.purs
file:
module SimpleService.Persistence
( insertUser
, findUser
, updateUser
, deleteUser
, listUsers
) where
import Prelude
import Control.Monad.Aff (Aff)
import Data.Array as Array
import Data.Maybe (Maybe)
import Database.PostgreSQL as PG
import SimpleService.Types (User(..), UserID)
insertUserQuery :: String
insertUserQuery = "insert into users (id, name) values ($1, $2)"
findUserQuery :: String
findUserQuery = "select id, name from users where id = $1"
updateUserQuery :: String
updateUserQuery = "update users set name = $1 where id = $2"
deleteUserQuery :: String
deleteUserQuery = "delete from users where id = $1"
listUsersQuery :: String
listUsersQuery = "select id, name from users"
insertUser :: forall eff. PG.Connection -> User
-> Aff (postgreSQL :: PG.POSTGRESQL | eff) Unit
insertUser conn user =
PG.execute conn (PG.Query insertUserQuery) user
findUser :: forall eff. PG.Connection -> UserID
-> Aff (postgreSQL :: PG.POSTGRESQL | eff) (Maybe User)
findUser conn userID =
map Array.head $ PG.query conn (PG.Query findUserQuery) (PG.Row1 userID)
updateUser :: forall eff. PG.Connection -> User
-> Aff (postgreSQL :: PG.POSTGRESQL | eff) Unit
updateUser conn (User {id, name}) =
PG.execute conn (PG.Query updateUserQuery) (PG.Row2 name id)
deleteUser :: forall eff. PG.Connection -> UserID
-> Aff (postgreSQL :: PG.POSTGRESQL | eff) Unit
deleteUser conn userID =
PG.execute conn (PG.Query deleteUserQuery) (PG.Row1 userID)
listUsers :: forall eff. PG.Connection
-> Aff (postgreSQL :: PG.POSTGRESQL | eff) (Array User)
listUsers conn =
PG.query conn (PG.Query listUsersQuery) PG.Row0
We can now write a simple HTTP API over the persistence layer using Express to provide CRUD functionality for users. Let’s install Express and purescript-express, the PureScript wrapper over it:
We do this top-down. First, we change src/Main.purs
to run the HTTP server by providing the server port and database configuration:
module Main where
import Prelude
import Control.Monad.Eff (Eff)
import Control.Monad.Eff.Console (CONSOLE)
import Database.PostgreSQL as PG
import Node.Express.Types (EXPRESS)
import SimpleService.Server (runServer)
main :: forall eff. Eff ( console :: CONSOLE
, express :: EXPRESS
, postgreSQL :: PG.POSTGRESQL
| eff) Unit
main = runServer port databaseConfig
where
port = 4000
databaseConfig = { user: "abhinav"
, password: ""
, host: "localhost"
, port: 5432
, database: "simple_service"
, max: 10
, idleTimeoutMillis: 1000
}
Next, we wire up the server routes to the handlers in src/SimpleService/Server.purs
:
module SimpleService.Server (runServer) where
import Prelude
import Control.Monad.Aff (runAff)
import Control.Monad.Eff (Eff)
import Control.Monad.Eff.Class (liftEff)
import Control.Monad.Eff.Console (CONSOLE, log, logShow)
import Database.PostgreSQL as PG
import Node.Express.App (App, get, listenHttp)
import Node.Express.Types (EXPRESS)
import SimpleService.Handler (getUser)
app :: forall eff. PG.Pool -> App (postgreSQL :: PG.POSTGRESQL | eff)
app pool = do
get "/v1/user/:id" $ getUser pool
runServer :: forall eff.
Int
-> PG.PoolConfiguration
-> Eff ( express :: EXPRESS
, postgreSQL :: PG.POSTGRESQL
, console :: CONSOLE
| eff ) Unit
runServer port databaseConfig = void $ runAff logShow pure do
pool <- PG.newPool databaseConfig
let app' = app pool
void $ liftEff $ listenHttp app' port \_ -> log $ "Server listening on :" <> show port
runServer
creates a PostgreSQL connection pool and passes it to the app
function which creates the Express application, which in turn, binds it to the handler getUser
. Then it launches the HTTP server by calling listenHttp
.
Finally, we write the actual getUser
handler in src/SimpleService/Handler.purs
:
module SimpleService.Handler where
import Prelude
import Control.Monad.Aff.Class (liftAff)
import Data.Foreign.Class (encode)
import Data.Int (fromString)
import Data.Maybe (Maybe(..))
import Database.PostgreSQL as PG
import Node.Express.Handler (Handler)
import Node.Express.Request (getRouteParam)
import Node.Express.Response (end, sendJson, setStatus)
import SimpleService.Persistence as P
getUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
getUser pool = getRouteParam "id" >>= case _ of
Nothing -> respond 422 { error: "User ID is required" }
Just sUserId -> case fromString sUserId of
Nothing -> respond 422 { error: "User ID must be an integer: " <> sUserId }
Just userId -> liftAff (PG.withConnection pool $ flip P.findUser userId) >>= case _ of
Nothing -> respond 404 { error: "User not found with id: " <> sUserId }
Just user -> respond 200 (encode user)
respond :: forall eff a. Int -> a -> Handler eff
respond status body = do
setStatus status
sendJson body
respondNoContent :: forall eff. Int -> Handler eff
respondNoContent status = do
setStatus status
end
getUser
validates the route parameter for valid user ID, sending error HTTP responses in case of failures. It then calls findUser
to find the user and returns appropriate response.
We can test this on the command-line using HTTPie. We run pulp --watch run
in one terminal to start the server with file watching, and test it from another terminal:
$ pulp --watch run
* Building project in ps-simple-rest-service
* Build successful.
Server listening on :4000
$ http GET https://localhost:4000/v1/user/1 # should return the user we created earlier
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 25
Content-Type: application/json; charset=utf-8
Date: Sun, 10 Sep 2017 14:32:52 GMT
ETag: W/"19-qmtK9XY+WDrqHTgqtFlV+h+NGOY"
X-Powered-By: Express
{
"id": 1,
"name": "Abhinav"
}
$ http GET https://localhost:4000/v1/user/s
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 38
Content-Type: application/json; charset=utf-8
Date: Sun, 10 Sep 2017 14:36:04 GMT
ETag: W/"26-//tvORl1gGDUMwgSaqbEpJhuadI"
X-Powered-By: Express
{
"error": "User ID must be an integer: s"
}
$ http GET https://localhost:4000/v1/user/2
HTTP/1.1 404 Not Found
Connection: keep-alive
Content-Length: 36
Content-Type: application/json; charset=utf-8
Date: Sun, 10 Sep 2017 14:36:11 GMT
ETag: W/"24-IyD5VT4E8/m3kvpwycRBQunI7Uc"
X-Powered-By: Express
{
"error": "User not found with id: 2"
}
deleteUser
handler is similar. We add the route in the app
function in the src/SimpleService/Server.purs
file:
-- previous code
import Node.Express.App (App, delete, get, listenHttp)
import SimpleService.Handler (deleteUser, getUser)
-- previous code
app :: forall eff. PG.Pool -> App (postgreSQL :: PG.POSTGRESQL | eff)
app pool = do
get "/v1/user/:id" $ getUser pool
delete "/v1/user/:id" $ deleteUser pool
-- previous code
And we add the handler in the src/SimpleService/Handler.purs
file:
deleteUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
deleteUser pool = getRouteParam "id" >>= case _ of
Nothing -> respond 422 { error: "User ID is required" }
Just sUserId -> case fromString sUserId of
Nothing -> respond 422 { error: "User ID must be an integer: " <> sUserId }
Just userId -> do
found <- liftAff $ PG.withConnection pool \conn -> PG.withTransaction conn do
P.findUser conn userId >>= case _ of
Nothing -> pure false
Just _ -> do
P.deleteUser conn userId
pure true
if found
then respondNoContent 204
else respond 404 { error: "User not found with id: " <> sUserId }
After the usual validations on the route param, deleteUser
tries to find the user by the given user ID and if found, it deletes the user. Both the persistence related functions are run inside a single SQL transaction using PG.withTransaction
function. deleteUser
return 404 status if the user is not found, else it returns 204 status.
Let’s try it out:
$ http GET https://localhost:4000/v1/user/1
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 25
Content-Type: application/json; charset=utf-8
Date: Mon, 11 Sep 2017 05:10:50 GMT
ETag: W/"19-GC9FAtbd81t7CtrQgsNuc8HITXU"
X-Powered-By: Express
{
"id": 1,
"name": "Abhinav"
}
$ http DELETE https://localhost:4000/v1/user/1
HTTP/1.1 204 No Content
Connection: keep-alive
Date: Mon, 11 Sep 2017 05:10:56 GMT
X-Powered-By: Express
$ http GET https://localhost:4000/v1/user/1
HTTP/1.1 404 Not Found
Connection: keep-alive
Content-Length: 37
Content-Type: application/json; charset=utf-8
Date: Mon, 11 Sep 2017 05:11:03 GMT
ETag: W/"25-Eoc4ZbEF73CyW8EGh6t2jqI8mLU"
X-Powered-By: Express
{
"error": "User not found with id: 1"
}
$ http DELETE https://localhost:4000/v1/user/1
HTTP/1.1 404 Not Found
Connection: keep-alive
Content-Length: 37
Content-Type: application/json; charset=utf-8
Date: Mon, 11 Sep 2017 05:11:05 GMT
ETag: W/"25-Eoc4ZbEF73CyW8EGh6t2jqI8mLU"
X-Powered-By: Express
{
"error": "User not found with id: 1"
}
createUser
handler is a bit more involved. First, we add an Express middleware to parse the body of the request as JSON. We use body-parser
for this and access it through PureScript FFI. We create a new file src/SimpleService/Middleware/BodyParser.js
with the content:
"use strict";
var bodyParser = require("body-parser");
exports.jsonBodyParser = bodyParser.json({
limit: "5mb"
});
And write a wrapper for it in the file src/SimpleService/Middleware/BodyParser.purs
with the content:
module SimpleService.Middleware.BodyParser where
import Prelude
import Data.Function.Uncurried (Fn3)
import Node.Express.Types (ExpressM, Response, Request)
foreign import jsonBodyParser ::
forall e. Fn3 Request Response (ExpressM e Unit) (ExpressM e Unit)
We also install the body-parser
npm dependency:
Next, we change the app
function in the src/SimpleService/Server.purs
file to add the middleware and the route:
-- previous code
import Node.Express.App (App, delete, get, listenHttp, post, useExternal)
import SimpleService.Handler (createUser, deleteUser, getUser)
import SimpleService.Middleware.BodyParser (jsonBodyParser)
-- previous code
app :: forall eff. PG.Pool -> App (postgreSQL :: PG.POSTGRESQL | eff)
app pool = do
useExternal jsonBodyParser
get "/v1/user/:id" $ getUser pool
delete "/v1/user/:id" $ deleteUser pool
post "/v1/users" $ createUser pool
And finally, we write the handler in the src/SimpleService/Handler.purs
file:
-- previous code
import Data.Either (Either(..))
import Data.Foldable (intercalate)
import Data.Foreign (renderForeignError)
import Node.Express.Request (getBody, getRouteParam)
import SimpleService.Types
-- previous code
createUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
createUser pool = getBody >>= case _ of
Left errs -> respond 422 { error: intercalate ", " $ map renderForeignError errs}
Right u@(User user) ->
if user.id <= 0
then respond 422 { error: "User ID must be positive: " <> show user.id}
else if user.name == ""
then respond 422 { error: "User name must not be empty" }
else do
liftAff (PG.withConnection pool $ flip P.insertUser u)
respondNoContent 201
createUser
calls getBody
which has type signature forall e a. (Decode a) => HandlerM (express :: EXPRESS | e) (Either MultipleErrors a)
. It returns either a list of parsing errors or a parsed instance, which in our case is a User
. In case of errors, we just return the errors rendered as string with a 422 status. If we get a parsed User
instance, we do some validations on it, returning appropriate error messages. If all validations pass, we create the user in the database by calling insertUser
from the persistence layer and respond with a status 201.
We can try it out:
$ http POST https://localhost:4000/v1/users name="abhinav"
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 97
Content-Type: application/json; charset=utf-8
Date: Mon, 11 Sep 2017 05:51:28 GMT
ETag: W/"61-BgsrMukZpImcdwAJEKCZ+70WBb8"
X-Powered-By: Express
{
"error": "Error at array index 0: (ErrorAtProperty \"id\" (TypeMismatch \"Int\" \"Undefined\"))"
}
$ http POST https://localhost:4000/v1/users id:=1 name=""
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 39
Content-Type: application/json; charset=utf-8
Date: Mon, 11 Sep 2017 05:51:42 GMT
ETag: W/"27-JQsh12xu/rEFdWy8REF4NMtBUB4"
X-Powered-By: Express
{
"error": "User name must not be empty"
}
$ http POST https://localhost:4000/v1/users id:=1 name="abhinav"
HTTP/1.1 201 Created
Connection: keep-alive
Content-Length: 0
Date: Mon, 11 Sep 2017 05:52:23 GMT
X-Powered-By: Express
$ http GET https://localhost:4000/v1/user/1
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 25
Content-Type: application/json; charset=utf-8
Date: Mon, 11 Sep 2017 05:52:30 GMT
ETag: W/"19-GC9FAtbd81t7CtrQgsNuc8HITXU"
X-Powered-By: Express
{
"id": 1,
"name": "abhinav"
}
First try returns a parsing failure because we didn’t provide the id
field. Second try is a validation failure because the name was empty. Third try is a success which we confirm by doing a GET
request next.
We want to allow a user’s name to be updated through the API, but not the user’s ID. So we add a new type to src/SimpleService/Types.purs
to represent a possible change in user’s name:
-- previous code
import Data.Foreign.NullOrUndefined (NullOrUndefined)
-- previous code
newtype UserPatch = UserPatch { name :: NullOrUndefined String }
derive instance genericUserPatch :: Generic UserPatch _
instance decodeUserPatch :: Decode UserPatch where
decode = genericDecode $ defaultOptions { unwrapSingleConstructors = true }
NullOrUndefined
is a wrapper over Maybe
with added support for Javascript null
and undefined
values. We define UserPatch
as having a possibly null (or undefined) name
field.
Now we can add the corresponding handler in src/SimpleService/Handlers.purs
:
-- previous code
import Data.Foreign.NullOrUndefined (unNullOrUndefined)
-- previous code
updateUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
updateUser pool = getRouteParam "id" >>= case _ of
Nothing -> respond 422 { error: "User ID is required" }
Just sUserId -> case fromString sUserId of
Nothing -> respond 422 { error: "User ID must be positive: " <> sUserId }
Just userId -> getBody >>= case _ of
Left errs -> respond 422 { error: intercalate ", " $ map renderForeignError errs}
Right (UserPatch userPatch) -> case unNullOrUndefined userPatch.name of
Nothing -> respondNoContent 204
Just userName -> if userName == ""
then respond 422 { error: "User name must not be empty" }
else do
savedUser <- liftAff $ PG.withConnection pool \conn -> PG.withTransaction conn do
P.findUser conn userId >>= case _ of
Nothing -> pure Nothing
Just (User user) -> do
let user' = User (user { name = userName })
P.updateUser conn user'
pure $ Just user'
case savedUser of
Nothing -> respond 404 { error: "User not found with id: " <> sUserId }
Just user -> respond 200 (encode user)
After checking for a valid user ID as before, we get the decoded request body as a UserPatch
instance. If the path does not have the name
field or has it as null
, there is nothing to do and we respond with a 204 status. If the user’s name is present in the patch, we validate it for non-emptiness. Then, within a database transaction, we try to find the user with the given ID, responding with a 404 status if the user is not found. If the user is found, we update the user’s name in the database, and respond with a 200 status and the saved user encoded as the JSON response body.
Finally, we can add the route to our server’s router in src/SimpleService/Server.purs
to make the functionality available:
-- previous code
import Node.Express.App (App, delete, get, http, listenHttp, post, useExternal)
import Node.Express.Types (EXPRESS, Method(..))
import SimpleService.Handler (createUser, deleteUser, getUser, updateUser)
-- previous code
app :: forall eff. PG.Pool -> App (postgreSQL :: PG.POSTGRESQL | eff)
app pool = do
useExternal jsonBodyParser
get "/v1/user/:id" $ getUser pool
delete "/v1/user/:id" $ deleteUser pool
post "/v1/users" $ createUser pool
patch "/v1/user/:id" $ updateUser pool
where
patch = http (CustomMethod "patch")
We can try it out now:
$ http GET https://localhost:4000/v1/user/1
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 26
Content-Type: application/json; charset=utf-8
Date: Fri, 11 Sep 2017 06:41:10 GMT
ETag: W/"1a-hoLBx55zeY8nZFWJh/kM05pXwSA"
X-Powered-By: Express
{
"id": 1,
"name": "abhinav"
}
$ http PATCH https://localhost:4000/v1/user/1 name=abhinavsarkar
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 31
Content-Type: application/json; charset=utf-8
Date: Fri, 11 Sep 2017 06:41:36 GMT
ETag: W/"1f-EG5i0hq/hYhF0BsuheD9hNXeBpI"
X-Powered-By: Express
{
"id": 1,
"name": "abhinavsarkar"
}
$ http GET https://localhost:4000/v1/user/1
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 31
Content-Type: application/json; charset=utf-8
Date: Fri, 11 Sep 2017 06:41:40 GMT
ETag: W/"1f-EG5i0hq/hYhF0BsuheD9hNXeBpI"
X-Powered-By: Express
{
"id": 1,
"name": "abhinavsarkar"
}
$ http PATCH https://localhost:4000/v1/user/1
HTTP/1.1 204 No Content
Connection: keep-alive
Date: Fri, 11 Sep 2017 06:42:31 GMT
X-Powered-By: Express
$ http PATCH https://localhost:4000/v1/user/1 name=""
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 39
Content-Type: application/json; charset=utf-8
Date: Fri, 11 Sep 2017 06:43:17 GMT
ETag: W/"27-JQsh12xu/rEFdWy8REF4NMtBUB4"
X-Powered-By: Express
{
"error": "User name must not be empty"
}
Listing all users is quite simple since it doesn’t require us to take any request parameter.
We add the handler to the src/SimpleService/Handler.purs
file:
-- previous code
listUsers :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
listUsers pool = liftAff (PG.withConnection pool P.listUsers) >>= encode >>> respond 200
And the route to the src/SimpleService/Server.purs
file:
-- previous code
import SimpleService.Handler (createUser, deleteUser, getUser, listUsers, updateUser)
-- previous code
app :: forall eff. PG.Pool -> App (postgreSQL :: PG.POSTGRESQL | eff)
app pool = do
useExternal jsonBodyParser
get "/v1/user/:id" $ getUser pool
delete "/v1/user/:id" $ deleteUser pool
post "/v1/users" $ createUser pool
patch "/v1/user/:id" $ updateUser pool
get "/v1/users" $ listUsers pool
where
patch = http (CustomMethod "patch")
And that’s it. We can test this endpoint:
$ http POST https://localhost:4000/v1/users id:=2 name=sarkarabhinav
HTTP/1.1 201 Created
Connection: keep-alive
Content-Length: 0
Date: Fri, 11 Sep 2017 07:06:24 GMT
X-Powered-By: Express
$ http GET https://localhost:4000/v1/users
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 65
Content-Type: application/json; charset=utf-8
Date: Fri, 11 Sep 2017 07:06:27 GMT
ETag: W/"41-btt9uNdG+9A1RO7SCLOsyMmIyFo"
X-Powered-By: Express
[
{
"id": 1,
"name": "abhinavsarkar"
},
{
"id": 2,
"name": "sarkarabhinav"
}
]
That concludes the first part of the two-part tutorial. We learned how to set up a PureScript project, how to access a Postgres database and how to create a JSON REST API over the database. The code till the end of this part can be found in github. In the next part, we’ll learn how to do API validation, application configuration and logging. Discuss this post in the comments.
If you liked this post, please leave a comment.
]]>The NRHM (National Rural Health Mission) is an initiative undertaken by the Government of India to address the health needs of under-served rural areas. The NRHM was initially tasked with addressing the health needs of 18 states that had been identified as having weak public health indicators.
In an effort to better this setup, the Government of India setup the NHSRC (National Health Systems Resource Centre), under the NRHM, to serve as an apex body for technical assistance. The goal of this institution is to improve health outcomes by facilitating governance reform, health systems innovations, and improved information sharing among all stakeholders — at the national, state, district and sub-district levels — through capacity development and convergence models.
One of the areas that the NHSRC works in is the provision of universal healthcare services. Universal access to good quality services — services that are effective, that are safe and satisfying to the patient; services that are patient and community centred, and services that make efficient use of the limited resources available.
The approach for achieving these objectives involves ensuring that every single health facility is scored against pre-defined standards with periodic supportive supervision for ensuring continual improvement.
There are three main components to conducting an assessment,
- Facilities
- Assessments themselves, and
- Facilitators
Facilitators conduct Assessments for Facilities.
Facilities are hospitals or other healthcare systems that exist at different levels,
Based on their level, facilities would be eligible for a particular type of assessment.
There are two main types of assessments in the system,
Assessments are a set of pre-defined standards against which facilities are scored. Assessments happen periodically throughout the year under supervision to ensure continual improvement.
Facilitators are people with varied experience, who are picked based on the type of assessment. For instance, National level assessments would require facilitators with a certain set of requirements as compared to a State level assessments. For instance, national level facilitators will consist of representatives from the programme divisions (maternal health, child health, family planning, etc.) of the Ministry of Health and Family Welfare, Government of India and National Health Systems Resource Centre.
The assessment process requires Facilitators to give a Score (of 0, 1 or 2) to a Measurable Element (ME).
Each ME belongs to an Area of Concern (AoC) and an AoC belongs to a Standard.
Standards: These are broad thematic areas with respect to cleanliness & hygiene, and can be termed as the “pillars” of the system.
Area of Concern: Each theme (Standard), has a fixed number of criteria that cover specific attributes.
Measurable Element: Is the lowest and most tangible unit of an assessment. MEs are specific requirements that the facilitators are expected to look for in a facility for ascertaining the extent of compliance and award a score.
Checkpoints: In the NQAS type assessment, MEs are broken down further into Checkpoints. Checkpoints are departmental checklists.
This is how the above-mentioned elements exist in the Assessments,
Assessments for Facilities are done via Checklists, which encapsulate all of the above (Standards, AoCs, MEs and Checkpoints).
Facilitators were handed sheets of paper with all the Standards, AoC and ME on them. Facilitators would traverse back and forth through this list, assign a compliance score to a ME until they were done with all AoC and subsequently with all Standards.
Advantages of the offline system,
Disadvantages of the the offline system,
Based on the disadvantages mentioned above, Nikhil (senior member of the NHSRC) thought it’d be best to use technology to make the process of conducting and syncing assessments easier. Gunak is the app that aimed at doing this.
Minimum feature list we aimed to launch the app with,
The challenges we faced,
Usage: For an assessment to be complete, facilitators have to finish all MEs across all AoCs.
Usage Pattern: Facilitators, physically visit facilities to conduct assessments. The offline medium sometimes forced the facilitators to finish Measurable Elements in a sequential order (as displayed on paper) since it would be easier to keep track of the progress of the assessment.
Notes:
We wanted to design a system that would be flexible and not dictate facilitator behaviour. A system that would always keep the facilitator aware of where they are and their progress. In addition to this, we had to ensure that readability and usability were the main focus since facilitators moved around a lot while conducting assessments.
The bottom navigation for an ME, allows facilitators to know the progress of the AoC these MEs belong to. It works as a navigation system where facilitators can jump forward to answer an ME and come back to answer another ME.
We wanted to find an equivalent for the ‘flip & find’ function, that facilitators perform on paper, for the app.
Search allows facilitators to find MEs, AoCs and Standards.
Our Search feature isn’t a simple text match. As seen below, it can identify that the word ‘hygiene’, could pertain to a Standard, an AoC or MEs.
One of the main drawbacks of the offline medium was its inability to generate reports. A facilitator or their superior would spend hours going through an assessment and assign a score for the same.
We took advantage of technology to provide detailed reports based on Departments, AoC and Score, which can be shared as an excel sheet or as an image.
This project would not have been possible if it wasn’t for the amazing team at Samanvay. This is our second project with them, you can read about the first one here,
Designing for Rural India — Part 1
The Gunak app is being rolled out in phases. Its first launch was in August 2017 by the Ministry of Health and Family Welfare, the app has a 4.8* rating on the play store.
Thank you so much for reading this. If you found this interesting, don’t forget to 👏 👏
If you have questions or thoughts about the post and/or would like to reach out to us outside of Medium, send us an email, moshimoshi@nilenso.com
🙏
Gunak — Designing with the Indian Government was originally published in uxdesign.cc on Medium, where people are continuing the conversation by highlighting and responding to this story.
]]>Journey of designing OpenCHS with Samanvay
India does not have a National health insurance or universal health care system for all its citizens. This has propelled the private sector to its dominant position in the healthcare market. Private companies provide the lion’s share of healthcare services in the country.
Despite the structures in place to ensure equality and funding from government and non-government sources, we still observe a visible gap in access to healthcare in rural areas compared to cities. A staggering 68% of the population lives in rural areas and has no or limited access to hospitals and clinics. Consequently, the rural population mostly relies on alternative medicine and government programmes.
If one had to paint a picture of rural India, it would be people living in mud houses in small villages. These villages have next to no electricity supply. There are setups of solar power stations which people use to charge their cellphones. Travelling to these villages is not always easy. One would require to change at least three different modes of transport to reach many of these villages. The residents seek out healthcare in cases of pregnancy and severe illness like Tuberculosis, Dengue, etc. However, villagers often do not seek treatment for early symptoms which appear less dangerous. This is not always due to lack of awareness. It is often also the case that infrastructure to support the long tail of healthcare is simply not that accessible in rural India.
The public healthcare in rural areas has been developed as a three-tier structure based on predetermined population norms.
Community Health Centres form the uppermost tier and are established and maintained by the State Government. Community health centres are staffed by four medical professionals supported by twenty-one paramedical and other staff. Surgeons, physicians, gynaecologists and paediatricians provide comprehensive care in each CHC.
Each CHC has thirty indoor beds, an Operating Theatre, X-ray, Labour Room, and Laboratory facilities. The community health centre provides expert facilities in obstetric and other care for patients referred to them by the four primary health centres within their jurisdiction.
Primary Health Centres (PHCs) comprise the second tier in rural healthcare. PHCs provide integrated, curative and preventive healthcare to the rural population with emphasis on preventive and promotive aspects. Activities include promotion of better health and hygiene practices, tetanus inoculation of pregnant women, intake of IFA tablets and institutional deliveries. A medical officer is in charge of the PHC supported by fourteen paramedical and other staff. Each primary health centre has four to six beds. Patients are referred to the PHCs by six sub-centres.
A sub-centre is the most peripheral institution and the first contact point between the primary healthcare system and the community. An Auxiliary Nurse Midwife (ANM) is in charge of six sub-centres each of which provides basic drugs for minor ailments. A sub-centre provides services in relation to maternal and child health, family welfare, nutrition, immunisation, diarrhoea control, and control of communicable diseases. ANMs also use Anmol Tablet, a product that aids in maintaining and collecting data, specific to primary health.
Apart from the government bodies there are other players in the system, which are closest to the villagers (bottom most in this tier). There are private hospitals, which run to provide special services or even basic healthcare.
Accessing health support is not that easy for village residents. They have to travel long distances to visit public hospitals spending money, and time that could be spent doing their daily chores. That’s when community health services come to play.
There are non-profit NGO Hospitals which run Community Health Services (CHS). They hire and train health workers who work closely with villagers in providing health services and education.
Health Workers who work as a part of the Community Health Services are known as VHWs (Village Health Workers), as a part of CRHP (Comprehensive Rural Health Programme). Jamkhed is a programme run under CRHP.
Jamkhed is centered around mobilising and building the capacity of the community, empowering the people to bring about their own improvements in health and poverty-alleviation. This is one of the better known and appreciated community health systems.
VHWs are individuals selected by the villagers. They are responsible for providing consultation and prescription on basic health care. Their appointment by the community helps establish villagers’ trust in them.
VHWs typically take care of a single village, but if required, they can be responsible for up to five villages. They are trained in basic healthcare, understanding symptoms and personality development. Since they are only trained in basic healthcare, they escalate severe cases and recommend the patients to visit hospitals. Quite often these volunteers are women.
Apart from this, VHWs also bust widely held superstitions by providing elementary health education.
Health workers use their considerable interpersonal communication skills to bring about important behavioural changes with respect to reproductive and hygiene practices in their rural communities.
The infrastructural support that these health workers get is mostly via Community Health programs and some inventory support from the government.
Here are some highlights about their daily operations -
The health workers still use paper for most of their work. They are provided with a chart that maps various symptoms to common ailments and relevant medication. While this may enable the health workers to swiftly treat common ailments, the strict mapping unfortunately severely restricts their ability to correctly identify edge cases or more subtle health issues.
Procurement of medicines is also done using paper. Health workers also track the need for medicines, availability and expiry dates of the medicines. Medicines are procured from Sub-centres where the health workers visit often to give reports to ANM’s about their respective villages.
The health workers aren’t equipped to look at patients history at the time of consultation. They keep manual records which are too comprehensive to look into at the time of visits.
OpenCHS strives to fill this gap and provide a decision support system that helps a health worker perform diagnoses. The system also provides the health workers with the steps for treatment.
OpenCHS is an Open Community Health Service platform that works in collaboration with NGO Hospitals. OpenCHS is a mobile app that is used by Village Health Workers (VHW) in the field or in their clinics when patients visit them.
OpenCHS helps VHWs to record patient data, perform diagnoses and manage programme statuses. The app also enables the health workers to consider individual patients’ medical histories during their diagnoses.
Considering the challenges and constraints that we were working with, this is how we approached the design of the application.
We were aware that the user is not going to be connected to a network 98% of the time, so we made the app work offline. We store all data locally on the device. The app only connects to a server when synchonising data. Typically health workers visit a sub-centre or another place with internet connectivity monthly. Recorded data is reported to the hospitals as part of the synchronisation.
The application provides decision support based on the recorded data and suggests treatment. We built our algorithm in collaboration with medical practitioners and took into account common health procedures. The algorithm even helps the health worker identify emergencies and provide appropriate medical care. When a patient needs medical attention from a trained medical professional, the application suggests them to inform the patients of the details and visit a hospital.
The application will be implemented with multi-lingual support. For each implementation, there will be localisation for specialised medical terminology used by the residents of the villages. Currently, we have an implementation in Marathi, which can be downloaded from here.
Ease of use was our major focus while designing this application.
The application is yet to be released. The plan is to implement it at one village first and then scale accordingly.
We are looking forward to two types of feedback. After the implementation of the application, the health workers will be trained. At this stage will be able to get quick feedback about usability and other challenges.
The second feedback will have a rather longer loop, where the health workers will be using it in the field and on their return we will learn about how it went and get more qualitative feedback. Here is all the documentation, including designs of the project so far.
There has been much discussion about each flow and decision, in multiple small and large sessions that we will be sharing in detail shortly so that you can better understand our efforts. Keep on the lookout :)
Thanks to nilenso and Samanvay team for the opportunity to work on this interesting problem. I also appreciate the feedback from Kenneth & Trouble in better articulating this post.
Do get in touch with me (noopur@nilenso.com) for any further questions or details. Any feedback will be highly appreciated.
Noopur wrote this story to share knowledge and to help nurture the design community. All articles published on uxdesign.cc follow that same philosophy.
Designing for Rural India — Part 1 was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.
]]>We recently concluded annual reviews at nilenso, at the end of which, Sandy, who recently joined us, asked: “Why is our process the best we can have?”
This led to a lot of reminiscing about how reviews were conducted at other companies that some of us had worked at.
One example went like this: I ask people to review me. I don’t get to read what they said, or find out if they even bothered to write anything at all. Someone from my team (a senior, usually my boss) would look at all these reports along with HR, and hold a meeting with me. The meeting basically involved my boss giving me a summary of the above, based on his/her understanding/interpretation. While I could ask for explanations and examples when I received feedback that I couldn’t relate to, they were not always forthcoming.
There was in fact a level system based on which variable pay was decided, but this was not disclosed. Since the bonuses were kept confidential, even if a person did find out what someone else made, they couldn’t question their pay relative to their colleague (maybe s/he is a better negotiator?).** There was no incremental movement between levels. Post x number of years, you either moved up or out.
Another colleague said that reviews at his previous organisation involved filling out a report, mentioning the number of bugs you had fixed (or caused), SLAs you had met (or missed), and so on, and that “regardless of all this, your appraisal merely depended on the ‘relationship’ you had developed with your manager. Nothing really came about even when you rejected whatever level the manager gave you, because in the end s/he decides it.”
While these may sound like extreme cases, they are very real (and not all that uncommon). Regardless of experience, the adjectives thrown around to describe the performance management process ranged from “random” and “unfair” to “opaque” and “senseless”.
We’ve written about how we conduct reviews before, but here’s a quick recap.
Stacey Adams’ equity theory, that describes how individuals perceive the distribution of resources is well documented. In order to apply this concept, however, and to determine whether one’s compensation is fair, what one needs is data on how the available resources (profits) were distributed to begin with. The approach that most organisations take in this regard is to hold back information relevant to employees. We run against that grain here.
I don’t claim that our method is perfect. In fact, it’s constantly being tweaked. I would posit, however, that the fact that the rationale behind it is completely transparent, and open to critique, makes it better than a system which keeps hidden from its participants, the thought that went into creating it.
I’m also not going to pretend that this will work for everyone. More specifically, I’ll admit that this process may not scale well, and beyond a point, will have to be adapted it to suit the needs of a company that employs hundreds, as opposed to tens of people.
In the meantime, I’ll leave you with this quote by JFK:
“We are not afraid to entrust the American people with unpleasant facts, foreign ideas, alien philosophies, and competitive values. For a nation that is afraid to let its people judge the truth and falsehood in an open market is a nation that is afraid of its people.”
**We would write down our bonuses on a piece of paper, throw them in a hat, and then read them out loud. This way, everyone knew where s/he stood, but no one knew who got what.
]]>Salaries: We are currently based out of Bangalore, India and it’s always a terrible idea to evaluate anything here on the basis of “averages”. We do, however, try to pay competitively but also never forget that just money is simply not enough.
Transparent finances: We pride ourselves on being a completely open company, in every way. Everyone at nilenso collectively decides which projects we take on, and what our expenses for the year should be. We go out of our way to make sure everyone in the company understands how much money we’re making and how we’re spending it. Our books of accounts and finances are available for anyone within the company to take a look, at any time. If something’s confusing, just ask.
Open reviews: Our review meetings that happen twice a year are completely open too, to anyone who wants to attend them. Everyone can read the reviews written by everyone else, and they’re all open to questioning.
Read more about our review / salary structure here.
Insurance: Medical care is expensive, so all our employees and their family — parents, children and spouses are covered by a comprehensive insurance policy (this means cashless, all ailments covered, pre-existing conditions included, no bureaucracy policy), where nilenso pays 100% of the premium.
Education allowance: While we don’t have a formal limit on what you can spend on education, employees are free to use the company credit card to buy books — technical or otherwise as well as pay for courses online or offline. We even have an English tutor who comes in every week to help our Operations staff improve their language skills. One of them is now giving his board examinations, after being out of school for more than a decade.
We also try to host educational events at the office — from Haskell lessons to technical talks (if you haven’t been to XTC — eXtreme Tuesday’s Club, ask us for an invite). We want you to attend conferences, whether that’s around the block or halfway around the world. Since we’ve been up and running, we’ve attended a ton of conferences: Euroclojure, Clojure/West, Clojure/conj, RubyConf San Diego, RubyConf India, React Conf SF, StrangeLoop, GCRC, Fifth Elephant, JSFoo, rootconf, pgconf, 50p — and that’s not an exhaustive list.
100% coverage on hardware / software: Whatever hardware, software, or services you need to do your job are always 100% on us. No red tape, no questions asked.
Ergonomic Furniture: If you’re not comfortable in the office, we’re not happy. If your desk is too high, or the lighting in your room is too bright, let us know, and we’ll do something about it.
Expense Account: Everyone who works at nilenso has access to our debit and credit cards to pay for any work-related expenses — software, hardware, travel, office supplies, books. If you’ve paid for something from your own pocket, that’s fine too! Just send us a photo of the receipt and we’ll reimburse it promptly (we have an app called kulu that keeps track of all of this). When you’re traveling abroad, we get you a prepaid forex card, so you don’t have to pay for anything expensive yourself. Basically, everything goes, as long as you’re reasonable (nobody has been unreasonable so far).
Fully stocked healthy pantry: Our pantry is restocked daily with fresh fruits, yoga bars, and healthy alternatives to aerated soft drinks (just put whatever you’d like on the list we have put up, and it shall magically appear in the kitchen). We also make sure there’s fresh milk, bread and cereal, for the days when you miss breakfast.
Lunches: We get lunch in the office everyday and eat out once a week (admittedly, this could be somewhat healthier than it is now — we order in, based on what people like, but we’re working on it).
Menstruation Leave : nilenso offers paid menstrual leave for anyone who needs it, no questions asked.
Vacations and Paid Holidays: nilenso offers 29 days of paid vacation and a few national holidays. Obviously, this doesn’t include times when you’re very ill — and when you are, we’d like you to take as much time as you need to recuperate before getting back to work. None of this is strictly monitored, so we’re often asked what happens if people take advantage of our liberal vacation policy. For the record, we don’t know, because it hasn’t happened.
Maternity and Paternity Leave: At nilenso, both parents are offered 6 months paid and a further 6 months unpaid leave around the time that they welcome a newborn.
Work: Being a technology company, we’re really passionate about deep tech, but also about education, healthcare, maps, renewable energy and a host of other subjects. The first couple of years of our journey at nilenso were spent trying to establish ourselves as a firm that could be counted on to deliver on technically challenging projects. Having done that, we’re now fortuitously positioned to use technology to solve real world problems. While this isn’t technically a benefit, we actively try to find projects that align with your interests, even if it comes at a cost.
Working weeks: We’re a small company, so people often end up doing way more than they would in other, larger firms — recruiting, sales calls, ordering lunch, talking to lawyers, taking Haskell lessons. We encourage 40-hour working weeks, and let our clients know what to expect as well.
Working remotely: Many ensonians often work from home (some more often than others). Some time last year, Tim even worked from a beachside getaway in Kerala for a couple of weeks. If there’s any way we can support you while you’re away, we do it. We use Slack / email as much as possible so you’re connected to everything going on at work. Alternatively, if you need a data dongle because there’s no WiFi where you’re going, pick one up from the office. None of our employees are fully remote, so we’re not there yet, but we’re definitely on our way.
It’s never easy to pen down what a company does for its employees, because it’s so difficult to separate that from what it does for itself. In our case though, there’s no need for separation: as a co-op, the company *is* its employees, and everyone who works here owns her just as much as the next person.
— — — — —
*Okay, I admit it, I wrote this post in January 2016 and then sat on it for a year.