Tinybird, JSON & JavaScript's bigint

Luis Nell

Luis Nell

In a recent project, we had to build a realtime analytics dashboard, for visualizing traffic data from various sources.

Each source is identified by an ID. These IDs are rather big numbers, think of them like an IPv6 address.

Long Int
A large, whole, number in the quintillion range.

I would have never anticipated what we would encounter next.

Big numbers (anything that needs more than 8 bytes) are generally not a big (pun intended) problem in most programming languages.

In most…

Enter JavaScript

This project sends the entire sensor data into Tinybird, a really cool product. It is basically a hosted Clickhouse database. So, built for analytics data!

We collect the data, process it a little bit and then send it off to an endpoint of the Tinybird REST API, as JSON.

The one requirement in this: the processing part has to be in JavaScript (Node LTS).

And this is where things become,… tricky.

JSON can't handle it

In recent years, JS has received native support for arbitrarily large numbers. Regular numbers are Number and anything that doesn't fit into 64 bit is BigInt.

So my assumption was: when we send or receive a huge number, JS should be able to handle that correctly.

It does not.

If we receive:

{"source": 55878094393440482840222217111980736512}

and I put that through a JSON.parse

> JSON.parse('{"source": 55878094393440482840222217111980736512}')
{ source: 5.587809439344048e+37 }

We can see that the native JSON parse throws away everything after 5587809439344048. (I tried that in browser, node and deno)

stringify also lacks support for this:

// The "n" at the end here signals to the interpreter to use BigInt:
> typeof 55878094393440482840222217111980736512n
'bigint'
> JSON.stringify({source: 55878094393440482840222217111980736512n})
Uncaught TypeError: Do not know how to serialize a BigInt
    at JSON.stringify (<anonymous>)

I am very sure there are good reasons for this. Nevertheless, this is incredibly frustrating and a source of errors if not properly verified.

The solution

Thankfully, the JavaScript ecosystem has come up with workarounds for this. In typical JS fashion, there are quite a few packages that get the job done.

I settled our problem with safe-stable-stringify and lossless-json (for parsing) because I liked the download counts and benchmarks.

> const stringify = require("safe-stable-stringify")
> stringify({source: 55878094393440482840222217111980736512n})
'{"source":55878094393440482840222217111980736512}'

This right there is proof why you need to write tests. Without them, this would have slipped through.

Andere Beiträge