Originally published here.
\nHello again đđđ and welcome to part two of this series talking about how we build beampipe.io, a privacy-conscious and fun alternative to Google Analytics.
\nIn part one we talked about backend so naturally it's now time for us to turn to frontend. I'll start with a confessionâ-âI am not by skill or by inclination a frontend developer. Don't get me wrong, I've written a lot of frontend code over the years (perhaps I'm just in denial) but it's normally been about getting something built rather than more careful crafting. I'd say that's changed a little in the last few years as frontend tooling has improved drastically. Seriously, if you are a backend engineer looking to get your feet wet, no time like the present.
\nAnyway back to the subject, what tools are we using and why. Let's start with some easy decisions. It's probably not an understatement to say that I like static types. The stronger the type system the better. Particularly with web development, I find this drastically reduces the number of mistakes I make (typos in variable names or wrong usage of APIs being particularly common problems). So without a very compelling reason not to, I start every project with Typescript.
\nThe other easy pick for me is React. This is pragmatic more than principled. I'm sure Vue and Angular and Alpine.js are all great in their own ways but as a lone developer I've got enough plates spinning already so I pick something simple which I now know relatively well.
\n![\"The](\"/_next/static/media/lambdacube.9fa0b555.png\")
\nLike a lot of modern applications, Beampipe's UI is mostly a single page application (SPA). This means that your browser typically downloads a blob of javascript which dynamically manipulates the content of the webpage to show the UI and effect transitions between pages. This is a particularly good fit for Beampipe, which is interactive and data-heavy.
\ncreate-react-app is a fantastically easy way to get started with React. However, for beampipe I opted to use NextJS instead.
\nNextJS's main claim to fame is that it shakes up the SPA model a little by enabling server side rendering (SSR). This means NextJS can execute your React code server side, render the UI of your application and send this down to your browser as fully-formed HTML. This reduces the delay before your application is rendered on the client and also helps search engines index your content.
\nBut actually neither of these are very compelling things for Beampipe. We generally don't need Google to index an analytics dashboard and the rendering latency is also not a huge concern (particularly as the client will often request new data as the user interacts with the page). On the other hand, I've found NextJS really easy to write code in and the SSR capabilities are handy for some of the public pages. For instance NextJS is able to statically render the landing and authentication pages which is quite nice when I get the odd spike of traffic.
\nUsing NextJS means I do have to run another container, but since I'm already using Kubernetes, this isn't a big problem. That said, for someone who hasn't used NextJS before, I'd probably default to create-react-app and save some of the headache.
\nBack in part one, I mentioned that we're using GraphQL for the API along with urql on the client. urql is great. Whenever I need a more advanced feature, I reach for it and it's there without having to dig deep into the internals (as I sometimes found I had to with apollo-client). React hooks make it a joy to use. It's also well supported in NextJS via next-urql. This will talk to your API server side and thus pass the page to the client fully formed and ready to go.
\nAs I said, frontend development generally hasn't been a strong point for me. However, the analytics space is filled with some very beautiful UIs (plausible, fathom among others). To compete at all with them, I'd need to make beampipe at least passable from a design perspective. In practice, this means having fairly tight control over the design and not (as I have done previously) just shoving in a framework.
\nFor this reason, I picked TailwindCSS. Tailwind is a fairly new, low-level CSS framework which basically just gives you some utility classes for doing things like adding padding or setting fixed sizes or working with flexbox. This looks and sounds horrifying at first but it actually seems to produce pretty good results. You can use React to abstract components with standardised (but very customised) look and feel. It's also quite handy when you want to take inspiration from other sites (a growing number are using tailwind) đ.
\nA fringe benefit of choosing such a low level CSS framework is that it's really forced me to try to get my head around things like Flexbox and the various quirks of how browsers render and size elements on the page. I won't pretend to enjoy this but it has been worthwhile and I'd recommend any frontend-curious backend engineer give it a spin.
\n![\"Real](\"/_next/static/media/css.ef71ff45.png\")
\nProgrammers often fear things that we do not understand or which appear to behave unpredictably. I'm sure many of us nightmares (or even current experiences) of impossible to track down bugs or things that seemed to just make no sense. I certainly experience this occasionally but it's something that has become joyfully less frequent as I've gained experience. It's now quite rare for me to feel stressed or intimidated by a bug.
\nNot so with CSS. Magic incantations that I don't understand copied and pasted from StackOverflow. It scares me. The solution to this, quite naturally, is to knuckle down and learn it properly (kind of). I now find the fear and self-doubt is starting to recede and UI work, whilst still frustrating, has become more fun. It's also really liberating to see something you don't like in the UI and generally just be able to go in and fix it up right away.
\nThat's all for now. In the meantime do come have click around at beampipe.io or write to us hello@beampipe.io Cheers!
", "url": "https://alexsparrow.dev/articles/building-beampipe-part-2", "title": "Building beampipe part II: frontend", "summary": "Hello again đ and welcome to part two of this series talking about how we build beampipe.io, a privacy-conscious and fun alternative to Google Analytics. In part one we talked about backend soâŚ", "date_modified": "2020-08-21T00:00:00.000Z", "author": { "name": "Alex Sparrow" } }, { "id": "https://alexsparrow.dev/articles/how-we-build-beampipe", "content_html": "Originally published here.
\n![\"Dashboard](\"/_next/static/media/screenshot.58eef793.png\")
\nI wanted to kick this blog off with a discussion of some of the technical decisions we made while building beampipe. However, I guess I canât really get away without talking at all about what the product is so here goesâŚ
\nBeampipe is a simple, privacy-focussed web analytics tool. Itâs somewhat similar in concept to Fathom and Plausible but with some slightly different goals, particularly with regards to our future roadmap. Put simply, and most fundamentally, we think web analytics tools should be easy and fun to use. Secondly, we think that where possible, these tools should respect userâs privacy. We take the same cookieless approach as the two aforementioned tools and explicitly not that of Google Analytics. Thirdly and hopefully distinctively, we think web analytics tools should be powerful. They should give you useful, actionable insights. We feel that there is no tool on the market ticking all three of these boxes and that we can build it!
\nOk so now that Iâve kept our marketing department happy, how are we building it? Iâm going to start out by describing how a web analytics service works.
\nAll of the products I name-dropped in the preamble function in basically the same way. You drop a little (or not-so-little in the case of Google Analytics) chunk of Javascript onto your page. That Javascript will then make a connection to a backend service and phone home with a little blob of data describing (to various levels of detail) who the user is and what theyâre doing on your site. There are some variations here: for instance that you can forego Javascript in favour of an invisible image. But thatâs pretty muchthe size of it.
\nSo the first piece to any web analytics service is to build this tracking script and a suitable backend to process and store the data that it spits out. So what did we choose here?
\n![\"An](\"/_next/static/media/beampipe.be9f5378.jpeg\")
\nIâm a big fan of Kotlin, a relatively new programming language from the JetBrains. Kotlin is built upon the Java Virtual Machine (JVM), inheriting a huge base of libraries and frameworks as well as an extremely performant runtime. On top of this, and much more importantly from my perspective, it is extremely fun and productive to write code in. It mixes in just enough functional programming magic to make you feel clever and powerful without quite the learning curve, cognitive burden and multiple competing styles that comes with languages like Haskell (or to a lesser degree, Scala).
\nOn top of these strengths, Kotlin also has some very nice support for asynchronous programming using coroutines, which can be quite an advantage from two perspectives:
\nOn top of Kotlin, I had to pick a web framework. For this I chose Micronaut which is a fairly full-featured JVM-based web framework with some specific support for Kotlin. Whilst Iâve previously preferred less full-featured frameworks, Iâve actually found that, once through the learning curve, Micronaut gives you a lot of useful stuff out of the box without forcing you to use it (e.g. security/authentication, database connection pools, GraphQL).
\nSo here we come to perhaps the most important choice. Web analytics tools have some fairly specific needs from a database point of view and ones I shanât pretend to be an expert in. Write rates are high (at least once people are using the tool). So we need something that can support a fairly high sustained write rate (equivalent to the peak simultaneous traffic for all of the sites using the service). Secondly, we are dealing with timeseries data â something for which certain databases have specific support. Query patterns against the data are often going to be so-called OLAP queries â i.e. aggregations over various axes in the recorded data.
\nThere are a lot of potentially good database choices here: influxdb, clickhouse, bigquery, redshift etc. However, two concerns led our decision: low overall ops overhead and low cost (since we want to be able to offer a free tier for the service). For this reason, we chose TimescaleDB which is a plugin on top of the excellent PostgreSQL database. This means we only have to live with a single database (including our non-timeseries data) and letâs us get started quickly and at low cost. In future, we may well switch out for something else, but this has served us well so far.
\nHaving strong API support is part of the longer term vision for the product. We want our customers to be able to integrate deeply with their analytics data. We also want to build a fantastic frontend experience and do it quickly (the first version of the product was launched within a week).
\nThus we chose GraphQL. Why? Web analytics data is not well represented by a graph and thus doesnât really benefit from GraphQLâs namesake feature. On the other hand, weâve found it very useful for building rich APIs with more sophisticated querying capabilities. It is also strongly typed, meaning that users of the API can have some confidence in the shape of the data structures returned. Lastly, alongside frontend libraries like apollo-client or urql we have a very opinionated, simple and powerful way to hook our web interface up to backend queries. This makes development easy and code readable.
\n![\"Very](\"/_next/static/media/architecture.d7221b19.png\")
\nSo first an admission, despite the pronouns employed in this blog post, Beampipe is a single engineer at the moment â namely me! As a full stack engineer with a lot of ground to cover, automation is key. On the other hand, I have to pick my battles. Continuous Integration (CI) tooling is essential. I simply donât have time to be manually testing, building and uploading things every time I make a change, at least not if we want to maintain velocity on the product. But I also donât have time for the kind of build and deployment pipeline you might find on bigger engineering teams.
\nFor this reason, I build the backend code as a Docker image using GitHub Actions. The build itself is coordinated with Gradle and makes use of Googleâs handy jib plugin which produces small, cacheable docker images.
\nDeployment is another area where there were trade-offs to be made. Iâd love to use something like Heroku and really not have to worry about deploying the application. On the other hand, these tend to be quite cost prohibitive and not necessarily a good fit for how I wanted to build the app. For that reason, I picked Kubernetes on Scalewayâs hosted Kapsule service. Iâve used this for a few projects now and itâs been excellent.
\nCurrently the backend is deployed as a monolith but it should be fairly easy to split things up should we need to scale component individually (e.g. separating the data ingest service from the query layer).
\nSo that pretty much wraps it up on the backend pieces of the app. In the next article weâll discuss the frontend in a similar manner. Hit me up with questions or feedback on hello@beampipe.io Cheers!
", "url": "https://alexsparrow.dev/articles/how-we-build-beampipe", "title": "How we build Beampipe", "summary": "Beampipe is a simple, powerful, privacy-focussed web analytics service. We discuss our tech stack which allows us to deliver features rapidly and reliably.", "date_modified": "2020-08-11T00:00:00.000Z", "author": { "name": "Alex Sparrow" } }, { "id": "https://alexsparrow.dev/articles/async-therefore-i-am", "content_html": "Originally published here.
\nAt Quartic Technologies we spend a lot of our time building micro-services. Our flagship product, GroundControl, consists of about 6 different services (with another 5 or 6 satellites for integration with external databases and APIs). Weâve more-or-less standardised on the Dropwizard framework, which has overall been a good choice â itâs fast, pretty low on boilerplate, and opinionated enough about healthchecks, metrics, configuration and such that weâve so far avoided an endless spiral of bikeshedding on those topics.
\nOne thing that has left me a little envious when prospecting alternatives such as Ratpack is the promise of asynchronous IO (or âasyncâ as itâs often abbreviated). As it turns out, servicing requests asynchronously with Dropwizard is not only eminently possible but actually downright straightforward.
\nCuriously, despite the popularity of asynchronous web frameworks, there seems to be pretty scant mention of how to get Dropwizard to service requests asynchronously. A quick Google for âdropwizard asyncâ provided my first clue: https://jersey.java.net/documentation/latest/async.html. Getting started is surprisingly easy:
\n@Path("/resource")\npublic class HelloResource {\n @GET\n public void asyncHelloWorld(@Suspended final AsyncResponse asyncResponse) {\n new Thread(() -> {\n sleepUninterruptibly(1000, MILLISECONDS);\n asyncResponse.resume("Hello world!");\n }).start();\n }\n}\nSo as youâll see above, your async resource methods receive an AsyncResponse object annotated with @Suspended. Your code can now schedule some long running computation or IO operation and hand control back to Dropwizard immediately, freeing the request thread up for other things. Once you have a result, the AsyncResponse object lets you drop it back to the client.
An important part of GroundControl is the geospatial backend (internally we call it Weyl after the awesome physicist). One of this serviceâs jobs is to render vector tiles from our datastore and serve them up to the user. This involves a fairly costly sequence of events:
\nWhen the datasets become large, this pipeline can take several seconds to complete. Worse still, our client-side mapping layer (based on mapbox-gl) will often fire off a ton of requests simultaneously. Since this pipeline is compute bound, there is no point trying to process more tiles in parallel than we have cores. Moreover, weâd like to avoid filling up Dropwizardâs thread pool with a load of requests all pointlessly fighting for limited CPU time.
\nTo maintain responsiveness under load, we want to offload these computations to a fixed size thread pool, handing the request thread back to Dropwizard to use for other things.
\n![\"Threads.](\"/_next/static/media/threads.75856c96.webp\")
\nIt was a pretty simple job to switch our resource methods to use the @Suspended magic. With a few changes to the tile rendering pipeline to make it use a small, fixed-size thread pool we were handling tile requests fully asynchronously!
One piece worth noting here is that this change didnât come without cost â in particular it meant rewriting some of our resource tests to deal with the AsyncResponse object.
\nTo test things were working as intended, I spun up the Locust load-testing tool and simulated 500 users all requesting vector tiles from our backend. This VisualVM graph was particularly satisfying!
\n![\"<b>Before](\"/_next/static/media/before.b1f942d2.webp\")
\n![\"<b>After](\"/_next/static/media/after.b2dca22f.webp\")
\nAsynchronous web frameworks offer some pretty compelling performance improvements at the cost of a little code complexity and a slightly more difficult testing story. Weâve found this trade-off to be worthwhile for certain workloads, and Dropwizard lets us mix and match synchronous and asynchronous resource methods as we like.
\nIâm Alex, and I work at Quartic Technologies. We help companies use data to improve their field operations.
", "url": "https://alexsparrow.dev/articles/async-therefore-i-am", "title": "Async therefore I am", "summary": "Going async with Dropwizard", "date_modified": "2017-03-02T00:00:00.000Z", "author": { "name": "Alex Sparrow" } } ] }