Designing for Open Source Software

Mattermost → Relay

When you hear the words “open source” attached to a product, what image comes to mind? Do you imagine an office full of designers obsessing over every little pixel? Or do you imagine an army of alpha-nerds piecing together a slap-dash knockoff of some proprietary software?

We all believe open source software is badly designed because, historically, it was. (#NotAllOpenSourceSoftware)

Thirty years ago, in 1988, the very concept of a license or design language for software was far beyond the average person. A decade later, everything was still awful. Software barely worked. Your PC definitely had a virus on it. HTML had a <blink> tag. Installing Linux required weeks of patience and belief in oneself — the kind of conviction I would need to train for a marathon. If there is one canonical failure we can promote as the poster child of this dark era, it is OpenOffice. Or StarOffice or LibreOffice or Apache OpenOffice or OOo or AOO or whatever name you might know it by. When the very name of the product is this broken it’s unlikely to succeed as a brand.

Fast forward ten more years. As of 2008, Firefox was eating Microsoft’s lunch. Firefox was open source, fast, and standards-compliant. It was easy to like Firefox. But we didn’t like Firefox. We loved Firefox. Firefox had a brand before it was Firefox. A brand isn’t a snappy name and a cute logo. Your product is your brand and your product is dictated by its design.

In 2018, user expectations demand that design asymptotically approach perfection: your product better be open, compatible, clean, fast, and powerful. When nilenso first made the move from Slack to Mattermost, we kept all of this in mind. From our very first deployment of Mattermost it felt whole. We chose Mattermost because it was so very close to the design we wanted, we knew we could help bring it the rest of the way.

The beginning

The design problems that currently exist with Mattermost aren’t just visual but interaction and experience related. One of the first issues we took up was adding infinite scroll to the channel history. We couldn’t fathom why users should manually click to load every page of history. So we fixed it!

In the not-too-distant future we will add all the little nuanced details which transform Relay into software you absolutely love. At the moment, however, our aim is to solve the obvious problems our users have with the Mattermost/Relay experience. Like infinite scroll, these are the problems that left our early users shouting at us “ugh… how have you not fixed this yet?!”

Such problems stem from the pains caused by inconsistencies and visual discrepancies which hamper usability. It is surprising to many how quickly these seemingly small issues amount to death by a thousand papercuts. It is equally surprising how quickly software becomes a joy to use once you get the basics right.

We’ll start with the basics — icons and fonts — and then move on to the specifics of how we redesigned Mattermost as Relay, piece by piece.

Icons

When we moved from Slack to Mattermost the difference in icons used was stark. All icons used in Mattermost were inconsistent with respect to weights. They were standing out of the interface rather than blending in.

After further digging, we realised that Mattermost uses the free version of Font Awesome. Don’t get me wrong, I love the guys at Font Awesome; I even backed their kickstarter. But not all icons being available in the free weights (regular & solid) is quite a bummer.

We had three rules for selecting an icon set:

Icons must be open source
Icons need to be extensive
Icons need to be available in different styles (line and solid)

Allow me to interrupt with an aside. Technically, it is possible for us to build commercial artefacts into the Relay client by keeping them out of the source tree, even though Relay is AGPL3-licensed (as a derivative of the mattermost-webapp). Why, then, do we feel it is essential that the icons be open source? Relay lives at the intersection of the principles of Free Software and the license which governs it. Relay stands on the shoulders of giants — not only the Mattermost source code but Linux, GoLang, React, and hundreds of other projects. Here, between the spirit and the letter of Open Philosophies, exists a great deal of nuance. Cutting through this nuance is a root concept, however: “share alike.” If we take something from the free/open software community, what we return to that community should afford the next recipient the same freedoms we had. If we tweak Relay’s design until it is absolutely perfect but that involves changes which Mattermost cannot absorb upstream, we have failed them and every other project we have built ourselves upon. Thus, our icons must have an open license. — Steven Deobald

Thanks Steve. Keeping the above mentioned points in mind, there were three icon sets that made the final round of discussion:

We chose the Clarity icon set because of styles, consistency, and the icons being part of a design language system. Clarity is an open source design system created and maintained by the good folks at VMware. Since VMware has a dedicated team pushing continuous updates there is little fear the icon set will be abandoned or go stale.

Font

Our font selection followed the same rules:

Fonts must be open source
Fonts should be available in multiple weights

…along with these additions:

Fonts with large x-height for easy readability
Fonts with thick Regular weight so fonts will be crisp

We stress so much on font weights because Relay is, first and foremost, a messaging tool. In Relay, you spend most of your time reading and we want this experience to be as enjoyable as possible. Relay also has complete Markdown support and weights help establish hierarchy both within the app and within the messages themselves.

We had a few contenders which satisfied these requirements. PT Sans, Lato, and Noto Sans all have an enjoyable reading experience but after a lot of trial and error we chose Fira Sans.

Fira Sans is an incredible font. It has a healthy variety of weights and the compressed nature of the font holds the messaging interface together while keeping it crisp.

Did you know that Fira Sans was built on top of Erik Spiekermann’s commercial font FF Meta? And that Spiekermann was commissioned by the Mozilla Foundation! Read the whole story here

How Mattermost became Relay

Implementing the design basics improved the existing experience. But to make those changes truly delightful we need to fine-tune every aspect of the app. We have started with three major sections:

Left sidebar
Channel header
Main messaging interface

1. Left sidebar

The sidebar’s Team section allows users to switch between teams within organisations that they’re a part of. The current Mattermost implementation truncates the team name and adds a byline. The problem with this is that it does not add any value since hovering over the team opens a tooltip with the team name anyway. We removed the byline and the truncated text.

The sidebar’s Channel section felt mashed together. There is a clear hierarchy intended, which includes channel headers such as “Favorite Channels” and a distinction between active and inactive channels. We brought this hierarchy to the surface. Active channels have a heavy weight and inactive channels fade away with a lower opacity, despite the increased crispness of Fira Sans.

Grouped channels now have tighter spacing and groups themselves have larger spacing, allowing group titles to have a smaller font. All together, this makes the hierarchy obvious even at a glance.

2. Channel header

In the channel header we cleaned up the icons. Here are the issues we encountered with the icons:

Most icons weren’t of the same weight
Some icons were of a different size
Some icons had some positioning overrides to align them

The old icons for the channel header stood out, drawing attention away from the main chat area. We wanted them instead to blend into the design — unobtrusive but still obvious. To do this we removed the borders and switched the icon set to Clarity.

In addition to icons, we fixed the CSS for the header and right sidebar, getting rid of all funny margins and relative positioning. We also added consistent heights, paddings, and sizes for each of the icons. These fixes don’t just add polish to the UI; once merged, they will also ensure it is easier for anyone in the community, including us, to customise Mattermost.

3. Messaging interface

The messaging interface is where users spend the most time. The messaging interface is also where Mattermost has most of its design inconsistencies, which are most glaring in the basics: alignment, spacing between posts, font size, and other elements on the interface such as the reply box.

After we moved to Fira Sans we bumped up the font size, made sure the headings and names are bolded to establish hierarchy, and fixed alignment and spacing issues to deliver a crisper and more enjoyable reading experience.

Themes

Accessibility matters. We have light-sensitive coworkers and we’re sure you do, too. That’s why Relay comes with light and dark themes out of the box.

There are many Relay/Mattermost-compatible themes available already but we will pour all our energy into these two. They already make Relay beautiful and with every kaizen they will only get better.

Of course, we don’t restrict you to these two themes. Relay also comes with community themes and Relay’s entire colour scheme is fully customisable… not just the sidebar (cough Slack cough). We are working to make Relay 100% customisable through CSS so users have complete control over every aspect of the UI, if they want.

Putting it all together

By following a consistent design language for icons, font, and themes we ensure all our “pieces” are whole unto themselves. Putting them back together, the application itself becomes whole.

Mattermost has published a design challenge that encourages designers from all over the world to improve Mattermost (and, hence, Relay) readability and usability. Our redesign addresses this challenge.

Along with visual design changes, Relay also aims to significantly improve its user experience. Our public Trello board contains our plan for Relay’s future.

Future

“Design” is the 8th most popular tag on Medium. “Open Source” is ranked 290. Source code licenses will never be sexy for the same reason your city’s water utility company will never be. The water supply is inherently a part of your city’s design. Similarly, your freedom as a user is designed into software you buy. Your ownership of a product is but one more limb in the tree of design.

Open source software needs good design because good design respects you, the user.

❤

Designing for Open Source Software was originally published in The Relay on Medium, where people are continuing the conversation by highlighting and responding to this story.

Original post by Varun Pai - check out Stories by Varun Pai on Medium

Designing for open source software

We all believe open source software is badly designed because, historically, it was.

Mattermost → Relay

We all believe open source software is badly designed because, historically, it was. (#NotAllOpenSourceSoftware)

The beginning

We’ll start with the basics — icons and fonts — and then move on to the specifics of how we redesigned Mattermost as Relay, piece by piece.

Icons

We had three rules for selecting an icon set:

Icons must be open source
Icons need to be extensive
Icons need to be available in different styles (line and solid)

Allow me to interrupt with an aside. Technically, it is possible for us to build commercial artefacts into the Relay client by keeping them out of the source tree, even though Relay is AGPL3-licensed (as a derivative of the mattermost-webapp). Why, then, do we feel it is essential that the icons be open source? Relay lives at the intersection of the principles of Free Software and the license which governs it. Relay stands on the shoulders of giants — not only the Mattermost source code but Linux, GoLang, React, and hundreds of other projects. Here, between the spirit and the letter of Open Philosophies, exists a great deal of nuance. Cutting through this nuance is a root concept, however: “share alike.” If we take something from the free/open software community, what we return to that community should afford the next recipient the same freedoms we had. If we tweak Relay’s design until it is absolutely perfect but that involves changes which Mattermost cannot absorb upstream, we have failed them and every other project we have built ourselves upon. Thus, our icons must have an open license. — Steven Deobald

Thanks Steve. Keeping the above mentioned points in mind, there were three icon sets that made the final round of discussion:

Font

Our font selection followed the same rules:

Fonts must be open source
Fonts should be available in multiple weights

…along with these additions:

Fonts with large x-height for easy readability
Fonts with thick Regular weight so fonts will be crisp

We had a few contenders which satisfied these requirements. PT Sans, Lato, and Noto Sans all have an enjoyable reading experience but after a lot of trial and error we chose Fira Sans.

Fira Sans is an incredible font. It has a healthy variety of weights and the compressed nature of the font holds the messaging interface together while keeping it crisp.

Did you know that Fira Sans was built on top of Erik Spiekermann’s commercial font FF Meta? And that Spiekermann was commissioned by the Mozilla Foundation! Read the whole story here

How Mattermost became Relay

Implementing the design basics improved the existing experience. But to make those changes truly delightful we need to fine-tune every aspect of the app. We have started with three major sections:

Left sidebar
Channel header
Main messaging interface

1. Left sidebar

Grouped channels now have tighter spacing and groups themselves have larger spacing, allowing group titles to have a smaller font. All together, this makes the hierarchy obvious even at a glance.

2. Channel header

In the channel header we cleaned up the icons. Here are the issues we encountered with the icons:

Most icons weren’t of the same weight
Some icons were of a different size
Some icons had some positioning overrides to align them

The old icons for the channel header stood out, drawing attention away from the main chat area. We wanted them instead to blend into the design—unobtrusive but still obvious. To do this we removed the borders and switched the icon set to Clarity.

3. Messaging interface

Themes

Accessibility matters. We have light-sensitive coworkers and we’re sure you do, too. That’s why Relay comes with light and dark themes out of the box.

There are many Relay/Mattermost-compatible themes available already but we will pour all our energy into these two. They already make Relay beautiful and with every kaizen they will only get better.

Putting it all together

By following a consistent design language for icons, font, and themes we ensure all our “pieces” are whole unto themselves. Putting them back together, the application itself becomes whole.

Along with visual design changes, Relay also aims to significantly improve its user experience. Our public Trello board contains our plan for Relay’s future.

Future

Open source software needs good design because good design respects you, the user.

❤

Designing for open source software was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.

Original post by Varun Pai - check out Stories by Varun Pai on Medium

Writing a Simple REST Web Service in PureScript - Part 2

21 minute read

To recap, in the first part of this two-part tutorial, we built a simple JSON REST web service in PureScript to create, update, get, list and delete users, backed by a Postgres database. In this part we’ll work on the rest of the requirements:

validation of API requests.
reading the server and database configs from environment variables.
logging HTTP requests and debugging info.

But first,

Bugs!

What happens if we hit a URL on our server which does not exist? Let’s fire up the server and test it:

$ pulp --watch run

$ http GET http://localhost:4000/v1/random
HTTP/1.1 404 Not Found
Connection: keep-alive
Content-Length: 148
Content-Security-Policy: default-src 'self'
Content-Type: text/html; charset=utf-8
Date: Sat, 30 Sep 2017 08:23:20 GMT
X-Content-Type-Options: nosniff
X-Powered-By: Express

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /v1/random</pre>
</body>
</html>

We get back a default HTML response with a 404 status from Express. Since we are writing a JSON API, we should return a JSON response in this case too. We add the following code in the src/SimpleService/Server.purs file to add a catch-all route and send a 404 status with a JSON error message:

-- previous code
import Data.Either (fromRight)
import Data.String.Regex (Regex, regex) as Re
import Data.String.Regex.Flags (noFlags) as Re
import Node.Express.App (App, all, delete, get, http, listenHttp, post, useExternal)
import Node.Express.Response (sendJson, setStatus)
import Partial.Unsafe (unsafePartial)
-- previous code

allRoutePattern :: Re.Regex
allRoutePattern = unsafePartial $ fromRight $ Re.regex "/.*" Re.noFlags

app :: forall eff. PG.Pool -> App (postgreSQL :: PG.POSTGRESQL | eff)
app pool = do
  useExternal jsonBodyParser

  get "/v1/user/:id"    $ getUser pool
  delete "/v1/user/:id" $ deleteUser pool
  post "/v1/users"      $ createUser pool
  patch "/v1/user/:id"  $ updateUser pool
  get "/v1/users"       $ listUsers pool

  all allRoutePattern do
    setStatus 404
    sendJson {error: "Route not found"}
  where
    patch = http (CustomMethod "patch")

allRoutePattern matches all routes because it uses a "/.*" regular expression. We place it as the last route to match all the otherwise unrouted requests. Let’s see what is the result:

$ http GET http://localhost:4000/v1/random
HTTP/1.1 404 Not Found
Connection: keep-alive
Content-Length: 27
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 08:46:46 GMT
ETag: W/"1b-772e0u4nrE48ogbR0KmKfSvrHUE"
X-Powered-By: Express

{
    "error": "Route not found"
}

Now we get a nicely formatted JSON response.

Another scenario is when our application throws some uncaught error. To simulate this, we shut down our postgres database and hit the server for listing users:

$ http GET http://localhost:4000/v1/users
HTTP/1.1 500 Internal Server Error
Connection: keep-alive
Content-Length: 372
Content-Security-Policy: default-src 'self'
Content-Type: text/html; charset=utf-8
Date: Sat, 30 Sep 2017 08:53:40 GMT
X-Content-Type-Options: nosniff
X-Powered-By: Express

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Error: connect ECONNREFUSED 127.0.0.1:5432<br> &nbsp; &nbsp;at Object._errnoException (util.js:1026:11)<br> &nbsp; &nbsp;at _exceptionWithHostPort (util.js:1049:20)<br> &nbsp; &nbsp;at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1174:14)</pre>
</body>
</html>

We get another default HTML response from Express with a 500 status. Again, in this case we’d like to return a JSON response. We add the following code to the src/SimpleService/Server.purs file:

-- previous code
import Control.Monad.Eff.Exception (message)
import Node.Express.App (App, all, delete, get, http, listenHttp, post, useExternal, useOnError)
-- previous code

app :: forall eff. PG.Pool -> App (postgreSQL :: PG.POSTGRESQL | eff)
app pool = do
  -- previous code
  useOnError \err -> do
    setStatus 500
    sendJson {error: message err}
  where
    patch = http (CustomMethod "patch")

We add the useOnError handler which comes with purescript-express to return the error message as a JSON response. Back on the command-line:

$ http GET http://localhost:4000/v1/users
HTTP/1.1 500 Internal Server Error
Connection: keep-alive
Content-Length: 47
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 09:01:37 GMT
ETag: W/"2f-cJuIW6961YCpo9TWDSZ9VWHLGHE"
X-Powered-By: Express

{
    "error": "connect ECONNREFUSED 127.0.0.1:5432"
}

It works! Bugs are fixed now. We proceed to add next features.

Validation

Let’s recall the code to update a user from the src/SimpleService/Handler.purs file:

updateUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
updateUser pool = getRouteParam "id" >>= case _ of
  Nothing -> respond 422 { error: "User ID is required" }
  Just sUserId -> case fromString sUserId of
    Nothing -> respond 422 { error: "User ID must be positive: " <> sUserId }
    Just userId -> getBody >>= case _ of
      Left errs -> respond 422 { error: intercalate ", " $ map renderForeignError errs}
      Right (UserPatch userPatch) -> case unNullOrUndefined userPatch.name of
        Nothing -> respondNoContent 204
        Just userName -> if userName == ""
          then respond 422 { error: "User name must not be empty" }
          else do
            savedUser <- liftAff $ PG.withConnection pool \conn -> PG.withTransaction conn do
              P.findUser conn userId >>= case _ of
                Nothing -> pure Nothing
                Just (User user) -> do
                  let user' = User (user { name = userName })
                  P.updateUser conn user'
                  pure $ Just user'
            case savedUser of
              Nothing -> respond 404 { error: "User not found with id: " <> sUserId }
              Just user -> respond 200 (encode user)

As we can see, the actual request handling logic is obfuscated by the request validation logic for the user id and the user name patch parameters. We also notice that we are using three constructs for validation here: Maybe, Either and if-then-else. However, we can use just Either to subsume all these cases as it can “carry” a failure as well as a success case. Either also comes with a nice monad transformer ExceptT which provides the do syntax for failure propagation. So we choose ExceptT as the base construct for our validation framework and write functions to upgrade Maybe and if-then-else to it. We add the following code to the src/SimpleService/Validation.purs file:

module SimpleService.Validation
  (module MoreExports, module SimpleService.Validation) where

import Prelude

import Control.Monad.Except (ExceptT, except, runExceptT)
import Data.Either (Either(..))
import Data.Maybe (Maybe(..))
import Node.Express.Handler (HandlerM, Handler)
import Node.Express.Response (sendJson, setStatus)
import Node.Express.Types (EXPRESS)
import Control.Monad.Except (except) as MoreExports

type Validation eff a = ExceptT String (HandlerM (express :: EXPRESS | eff)) a

exceptMaybe :: forall e m a. Applicative m => e -> Maybe a -> ExceptT e m a
exceptMaybe e a = except $ case a of
  Just x  -> Right x
  Nothing -> Left e

exceptCond :: forall e m a. Applicative m => e -> (a -> Boolean) -> a -> ExceptT e m a
exceptCond e cond a = except $ if cond a then Right a else Left e

withValidation :: forall eff a. Validation eff a -> (a -> Handler eff) -> Handler eff
withValidation action handler = runExceptT action >>= case _ of
  Left err -> do
    setStatus 422
    sendJson {error: err}
  Right x  -> handler x

We re-export except from the Control.Monad.Except module. We also add a withValidation function which runs an ExceptT based validation and either returns an error response with a 422 status in case of a failed validation or runs the given action with the valid value in case of a successful validation.

Using these functions, we now write updateUser in the src/SimpleService/Handler.purs file as:

-- previous code
import Control.Monad.Trans.Class (lift)
import Data.Bifunctor (lmap)
import Data.Foreign (ForeignError, renderForeignError)
import Data.List.NonEmpty (toList)
import Data.List.Types (NonEmptyList)
import Data.Tuple (Tuple(..))
import SimpleService.Validation as V
-- previous code

renderForeignErrors :: forall a. Either (NonEmptyList ForeignError) a -> Either String a
renderForeignErrors = lmap (toList >>> map renderForeignError >>> intercalate ", ")

updateUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
updateUser pool = V.withValidation (Tuple <$> getUserId <*> getUserPatch)
                                   \(Tuple userId (UserPatch userPatch)) ->
    case unNullOrUndefined userPatch.name of
      Nothing -> respondNoContent 204
      Just uName -> V.withValidation (getUserName uName) \userName -> do
        savedUser <- liftAff $ PG.withConnection pool \conn -> PG.withTransaction conn do
          P.findUser conn userId >>= case _ of
            Nothing -> pure Nothing
            Just (User user) -> do
              let user' = User (user { name = userName })
              P.updateUser conn user'
              pure $ Just user'
        case savedUser of
          Nothing -> respond 404 { error: "User not found with id: " <> show userId }
          Just user -> respond 200 (encode user)
  where
    getUserId = lift (getRouteParam "id")
      >>= V.exceptMaybe "User ID is required"
      >>= fromString >>> V.exceptMaybe "User ID must be positive"

    getUserPatch = lift getBody >>= V.except <<< renderForeignErrors

    getUserName = V.exceptCond "User name must not be empty" (_ == "")

The validation logic has been extracted out in separate functions now which are composed using Applicative. The validation steps are composed using the ExceptT monad. We are now free to express the core logic of the function clearly. We rewrite the src/SimpleService/Handler.purs file using the validations:

module SimpleService.Handler where

import Prelude

import Control.Monad.Aff.Class (liftAff)
import Control.Monad.Trans.Class (lift)
import Data.Bifunctor (lmap)
import Data.Either (Either)
import Data.Foldable (intercalate)
import Data.Foreign (ForeignError, renderForeignError)
import Data.Foreign.Class (encode)
import Data.Foreign.NullOrUndefined (unNullOrUndefined)
import Data.Int (fromString)
import Data.List.NonEmpty (toList)
import Data.List.Types (NonEmptyList)
import Data.Maybe (Maybe(..))
import Data.Tuple (Tuple(..))
import Database.PostgreSQL as PG
import Node.Express.Handler (Handler)
import Node.Express.Request (getBody, getRouteParam)
import Node.Express.Response (end, sendJson, setStatus)
import SimpleService.Persistence as P
import SimpleService.Validation as V
import SimpleService.Types

getUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
getUser pool = V.withValidation getUserId \userId ->
  liftAff (PG.withConnection pool $ flip P.findUser userId) >>= case _ of
    Nothing -> respond 404 { error: "User not found with id: " <> show userId }
    Just user -> respond 200 (encode user)

deleteUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
deleteUser pool = V.withValidation getUserId \userId -> do
  found <- liftAff $ PG.withConnection pool \conn -> PG.withTransaction conn do
    P.findUser conn userId >>= case _ of
      Nothing -> pure false
      Just _  -> do
        P.deleteUser conn userId
        pure true
  if found
    then respondNoContent 204
    else respond 404 { error: "User not found with id: " <> show userId }

createUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
createUser pool = V.withValidation getUser \user@(User _) -> do
  liftAff (PG.withConnection pool $ flip P.insertUser user)
  respondNoContent 201
  where
    getUser = lift getBody
      >>= V.except <<< renderForeignErrors
      >>= V.exceptCond "User ID must be positive" (\(User user) -> user.id > 0)
      >>= V.exceptCond "User name must not be empty" (\(User user) -> user.name /= "")

updateUser :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
updateUser pool = V.withValidation (Tuple <$> getUserId <*> getUserPatch)
                                   \(Tuple userId (UserPatch userPatch)) ->
    case unNullOrUndefined userPatch.name of
      Nothing -> respondNoContent 204
      Just uName -> V.withValidation (getUserName uName) \userName -> do
        savedUser <- liftAff $ PG.withConnection pool \conn -> PG.withTransaction conn do
          P.findUser conn userId >>= case _ of
            Nothing -> pure Nothing
            Just (User user) -> do
              let user' = User (user { name = userName })
              P.updateUser conn user'
              pure $ Just user'
        case savedUser of
          Nothing -> respond 404 { error: "User not found with id: " <> show userId }
          Just user -> respond 200 (encode user)
  where
    getUserPatch = lift getBody >>= V.except <<< renderForeignErrors
    getUserName = V.exceptCond "User name must not be empty" (_ /= "")

listUsers :: forall eff. PG.Pool -> Handler (postgreSQL :: PG.POSTGRESQL | eff)
listUsers pool = liftAff (PG.withConnection pool P.listUsers) >>= encode >>> respond 200

getUserId :: forall eff. V.Validation eff Int
getUserId = lift (getRouteParam "id")
  >>= V.exceptMaybe "User ID is required"
  >>= fromString >>> V.exceptMaybe "User ID must be an integer"
  >>= V.exceptCond "User ID must be positive" (_ > 0)

renderForeignErrors :: forall a. Either (NonEmptyList ForeignError) a -> Either String a
renderForeignErrors = lmap (toList >>> map renderForeignError >>> intercalate ", ")

respond :: forall eff a. Int -> a -> Handler eff
respond status body = do
  setStatus status
  sendJson body

respondNoContent :: forall eff. Int -> Handler eff
respondNoContent status = do
  setStatus status
  end

The code is much cleaner now. Let’s try out a few test cases:

$ http POST http://localhost:4000/v1/users id:=3 name=roger
HTTP/1.1 201 Created
Connection: keep-alive
Content-Length: 0
Date: Sat, 30 Sep 2017 12:13:37 GMT
X-Powered-By: Express

$ http POST http://localhost:4000/v1/users id:=3
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 102
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:13:50 GMT
ETag: W/"66-/c4cfoquQZGwtDBUzHjJydJAHJ0"
X-Powered-By: Express

{
    "error": "Error at array index 0: (ErrorAtProperty \"name\" (TypeMismatch \"String\" \"Undefined\"))"
}

$ http POST http://localhost:4000/v1/users id:=3 name=""
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 39
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:14:02 GMT
ETag: W/"27-JQsh12xu/rEFdWy8REF4NMtBUB4"
X-Powered-By: Express

{
    "error": "User name must not be empty"
}

$ http POST http://localhost:4000/v1/users id:=0 name=roger
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 36
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:14:14 GMT
ETag: W/"24-Pvt1L4eGilBmVtaOGHlSReJ413E"
X-Powered-By: Express

{
    "error": "User ID must be positive"
}

$ http GET http://localhost:4000/v1/user/3
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 23
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:14:28 GMT
ETag: W/"17-1scpiB1FT9DBu9s4I1gNWSjH2go"
X-Powered-By: Express

{
    "id": 3,
    "name": "roger"
}

$ http GET http://localhost:4000/v1/user/asdf
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 38
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:14:40 GMT
ETag: W/"26-//tvORl1gGDUMwgSaqbEpJhuadI"
X-Powered-By: Express

{
    "error": "User ID must be an integer"
}

$ http GET http://localhost:4000/v1/user/-1
HTTP/1.1 422 Unprocessable Entity
Connection: keep-alive
Content-Length: 36
Content-Type: application/json; charset=utf-8
Date: Sat, 30 Sep 2017 12:14:45 GMT
ETag: W/"24-Pvt1L4eGilBmVtaOGHlSReJ413E"
X-Powered-By: Express

{
    "error": "User ID must be positive"
}

It works as expected.

Configuration

Right now our application configuration resides in the main function:

main = runServer port databaseConfig
  where
    port = 4000
    databaseConfig = { user: "abhinav"
                     , password: ""
                     , host: "localhost"
                     , port: 5432
                     , database: "simple_service"
                     , max: 10
                     , idleTimeoutMillis: 1000
                     }

We are going to extract it out of the code and read it from the environment variables using the purescript-config package. First, we install the required packages using bower.

$ bower install --save purescript-node-process purescript-config

Now, we write the following code in the src/SimpleService/Config.purs file:

module SimpleService.Config where

import Data.Config
import Prelude

import Control.Monad.Eff (Eff)
import Data.Config.Node (fromEnv)
import Data.Either (Either)
import Data.Set (Set)
import Database.PostgreSQL as PG
import Node.Process (PROCESS)

type ServerConfig =
  { port           :: Int
  , databaseConfig :: PG.PoolConfiguration
  }

databaseConfig :: Config {name :: String} PG.PoolConfiguration
databaseConfig =
  { user: _, password: _, host: _, port: _, database: _, max: _, idleTimeoutMillis: _ }
  <$> string {name: "user"}
  <*> string {name: "password"}
  <*> string {name: "host"}
  <*> int    {name: "port"}
  <*> string {name: "database"}
  <*> int    {name: "pool_size"}
  <*> int    {name: "idle_conn_timeout_millis"}

portConfig :: Config {name :: String} Int
portConfig = int {name: "port"}

serverConfig :: Config {name :: String} ServerConfig
serverConfig =
  { port: _, databaseConfig: _}
  <$> portConfig
  <*> prefix {name: "db"} databaseConfig

readServerConfig :: forall eff.
                    Eff (process :: PROCESS | eff) (Either (Set String) ServerConfig)
readServerConfig = fromEnv "SS" serverConfig

We use the applicative DSL provided in Data.Config module to build a description of our configuration. This description contains the keys and types of the configuration, for consumption by various interpreters. Then we use the fromEnv interpreter to read the config from the environment variables derived from the name fields in the records in the description in the readServerConfig function. We also write a bash script to set those environment variables in the development environment in the setenv.sh file:

export SS_PORT=4000
export SS_DB_USER="abhinav"
export SS_DB_PASSWORD=""
export SS_DB_HOST="localhost"
export SS_DB_PORT=5432
export SS_DB_DATABASE="simple_service"
export SS_DB_POOL_SIZE=10
export SS_DB_IDLE_CONN_TIMEOUT_MILLIS=1000

Now we rewrite our src/Main.purs file to use the readServerConfig function:

module Main where

import Prelude

import Control.Monad.Eff (Eff)
import Control.Monad.Eff.Console (CONSOLE, log)
import Data.Either (Either(..))
import Data.Set (toUnfoldable)
import Data.String (joinWith)
import Database.PostgreSQL as PG
import Node.Express.Types (EXPRESS)
import Node.Process (PROCESS)
import Node.Process as Process
import SimpleService.Config (readServerConfig)
import SimpleService.Server (runServer)

main :: forall eff. Eff ( console :: CONSOLE
                        , express :: EXPRESS
                        , postgreSQL :: PG.POSTGRESQL
                        , process :: PROCESS
                        | eff ) Unit
main = readServerConfig >>= case _ of
  Left missingKeys -> do
    log $ "Unable to start. Missing Env keys: " <> joinWith ", " (toUnfoldable missingKeys)
    Process.exit 1
  Right { port, databaseConfig } -> runServer port databaseConfig

If readServerConfig fails, we print the missing keys to the console and exit the process. Else we run the server with the read config.

To test this, we stop the server we ran in the beginning, source the config, and run it again:

$ pulp --watch run
* Building project in /Users/abhinav/ps-simple-rest-service
* Build successful.
Server listening on :4000
^C
$ source setenv.sh
$ pulp --watch run
* Building project in /Users/abhinav/ps-simple-rest-service
* Build successful.
Server listening on :4000

It works! We test the failure case by opening another terminal which does not have the environment variables set:

$ pulp run
* Building project in /Users/abhinav/ps-simple-rest-service
* Build successful.
Unable to start. Missing Env keys: SS_DB_DATABASE, SS_DB_HOST, SS_DB_IDLE_CONN_TIMEOUT_MILLIS, SS_DB_PASSWORD, SS_DB_POOL_SIZE, SS_DB_PORT, SS_DB_USER, SS_PORT
* ERROR: Subcommand terminated with exit code 1

Up next, we add logging to our application.

Logging

For logging, we use the purescript-logging package. We write a logger which logs to stdout; in the src/SimpleService/Logger.purs file:

module SimpleService.Logger
  ( debug
  , info
  , warn
  , error
  ) where

import Prelude

import Control.Logger as L
import Control.Monad.Eff.Class (class MonadEff, liftEff)
import Control.Monad.Eff.Console as C
import Control.Monad.Eff.Now (NOW, now)
import Data.DateTime.Instant (toDateTime)
import Data.Either (fromRight)
import Data.Formatter.DateTime (Formatter, format, parseFormatString)
import Data.Generic.Rep (class Generic)
import Data.Generic.Rep.Show (genericShow)
import Data.String (toUpper)
import Partial.Unsafe (unsafePartial)

data Level = Debug | Info | Warn | Error

derive instance eqLevel :: Eq Level
derive instance ordLevel :: Ord Level
derive instance genericLevel :: Generic Level _

instance showLevel :: Show Level where
  show = toUpper <<< genericShow

type Entry =
  { level   :: Level
  , message :: String
  }

dtFormatter :: Formatter
dtFormatter = unsafePartial $ fromRight $ parseFormatString "YYYY-MM-DD HH:mm:ss.SSS"

logger :: forall m e. (
          MonadEff (console :: C.CONSOLE, now :: NOW | e) m) => L.Logger m Entry
logger = L.Logger $ \{ level, message } -> liftEff do
  time <- toDateTime <$> now
  C.log $ "[" <> format dtFormatter time <> "] " <> show level <> " " <> message

log :: forall m e.
        MonadEff (console :: C.CONSOLE , now :: NOW | e) m
     => Entry -> m Unit
log entry@{level} = L.log (L.cfilter (\e -> e.level == level) logger) entry

debug :: forall m e.
         MonadEff (console :: C.CONSOLE , now :: NOW | e) m => String -> m Unit
debug message = log { level: Debug, message }

info :: forall m e.
        MonadEff (console :: C.CONSOLE , now :: NOW | e) m => String -> m Unit
info message = log { level: Info, message }

warn :: forall m e.
        MonadEff (console :: C.CONSOLE , now :: NOW | e) m => String -> m Unit
warn message = log { level: Warn, message }

error :: forall m e.
         MonadEff (console :: C.CONSOLE , now :: NOW | e) m => String -> m Unit
error message = log { level: Error, message }

purescript-logging lets us define our own logging levels and loggers. We define four log levels, and a log entry type with the log level and the message. Then we write the logger which will print the log entry to stdout along with the current time as a well formatted string. We define convenience functions for each log level.

Before we proceed, let’s install the required dependencies.

$ bower install --save purescript-logging purescript-now purescript-formatters

Now we add a request logger middleware to our server in the src/SimpleService/Server.purs file:

-- previous code
import Control.Monad.Eff.Console (CONSOLE)
import Control.Monad.Eff.Now (NOW)
import Data.Maybe (maybe)
import Data.String (toUpper)
import Node.Express.App (App, all, delete, get, http, listenHttp, post, use, useExternal, useOnError)
import Node.Express.Handler (Handler, next)
import Node.Express.Request (getMethod, getPath)
import SimpleService.Logger as Log
-- previous code

requestLogger :: forall eff. Handler (console :: CONSOLE, now :: NOW | eff)
requestLogger = do
  method <- getMethod
  path   <- getPath
  Log.debug $ "HTTP: " <> maybe "" id ((toUpper <<< show) <$> method) <> " " <> path
  next

app :: forall eff.
       PG.Pool
    -> App (postgreSQL :: PG.POSTGRESQL, console :: CONSOLE, now :: NOW | eff)
app pool = do
  useExternal jsonBodyParser
  use requestLogger
  -- previous code

We also convert all our previous logging statements which used Console.log to use SimpleService.Logger and add logs in our handlers. We can see logging in effect by restarting the server and hitting it:

$ pulp --watch run
* Building project in /Users/abhinav/ps-simple-rest-service
* Build successful.
[2017-09-30 16:02:41.634] INFO Server listening on :4000
[2017-09-30 16:02:43.494] DEBUG HTTP: PATCH /v1/user/3
[2017-09-30 16:02:43.517] DEBUG Updated user: 3
[2017-09-30 16:03:46.615] DEBUG HTTP: DELETE /v1/user/3
[2017-09-30 16:03:46.635] DEBUG Deleted user 3
[2017-09-30 16:05:03.805] DEBUG HTTP: GET /v1/users

Conclusion

In this tutorial we learned how to create a simple JSON REST web service written in PureScript with persistence, validation, configuration and logging. The complete code for this tutorial can be found in github. This post can be discussed on r/purescript.

If you liked this post, please leave a comment.

Original post by Abhinav Sarkar - check out All posts on abhinavsarkar.net

← Older Blog Archives