Using JSON to Exchange Data

Javascript Object Notation (JSON) is a fancy name for a simple idea: A web page can download data stored as javascript variables. It’s caused a buzz in the tech world because JSON is much easier to load, read and manipulate compared to XML. Also, parsing large XML files can be a large performance hit – JSON gives you an object, packaged up and ready to go!

Remember the script tag?

In javascript you can import any script using notation like this:

<script src="http://www.mysite.com/myscript.js"></script>

The browser will pull down the script and run it inside the current page. This is how embeddable widgets like Adsense and InstaCalc work.

The neat thing is that any variables and functions defined inside the script are available to the page. If myscript.js includes a function showMeTheMoney(), your page can call it (and, presumably, get shown the money).

This is all fine and dandy, and been known for a while. A cool use is to provide dynamic data to a page with encoded Javascript variables. If the variables were “objects” and we passed them using their special “notation”, we could call the system JavaScript Object Notation. (Technically, JSON just covers the “object” part; executable functions are dynamic javascript).

In Javascript, you can use objects like this:

var fruit = {}; // create new obj
fruit.name = "apple"; // set properties
fruit.color = "red";

// that is old fashioned! Try my new notation:

var fancyFruit = {"name":"pear", "color": "greenish"};

Javascript has a nice notation where you can define objects using "key":"value" pairs. You can define arrays and functions this way too, so your objects are more like a class, in fact.

So why is this useful again?

So far, passing scripts around is old news. But an idea comes along: what if you load a dynamically generated script, which has data inside of it? Now you can access the data as javascript variables. Here’s what I mean:

<script src="http://www.weathersite.com/latestweather.js"></script>
Today's weather is:
<script>
document.write(Weather.today);
</script>

See what’s happening here? We load the javascript (which could be updated every hour by the server), and access the Weather object it defines. Presumably, Weather.today is a string containing today’s weather. Realistically, we could have Weather[zipcode] or whatever format the site defines.

Cool, huh?

But why not XML?

Ah — some programmers will say “Verily, can you not download an XML file, and then process and parse it into Javascript variables?”

Sure, just like you can run a marathon or floss your teeth. Many want to, some try to, not everybody does.

XML is fine for certain things, but it can be quite cumbersome. JSON is great because you can include data in an easy, painless process. There’s no parsing step – you are getting your variables “for free” by just including the javascript file.

Also, JSON lets the site include functions, which may be used to process the data or handle other tasks.

A few details (as always)

There are various ways to use JSON (see links for source of code sample):

• Include a script tag directly (as above). This means the code inside the script is run immediately as the browser encounters it.

<script src="http://www.mysite.com/mydata.js"></script>

• Dynamically load the script tag. You can add a javascript tag to the head of a document (from codehouse.com):

function dhtmlLoadScript(url)
{
   var e = document.createElement("script");
   e.src = url;
   e.type="text/javascript";
   document.getElementsByTagName("head")[0].appendChild(e); 
}
dhtmlLoadScript("http://www.mysite.com/mydata.js");

• Use AJAX (Xml Http Request) to pull some text from the server and then eval it. Note that the first two techniques automatically “eval” the javascript because it is inside a script tag. You probably want to use a library to do the XHR request (like AjaxCaller):

ajaxCaller.getPlainText("mydata.js", 
   function(jsText) { eval(jsText); });

Note that XHR requests can only open files on the same domain as the caller. The first two techniques can load a file from any domain.

Security, Security, Security

I’m not an expert in Web Security, but I know enough to realize it’s tricky and that there will be things I miss. Even the big boys like Google can get caught in bugs sometimes.

The safest approach is to only store public data with JSON. If you store private data in JSON (or XML for that matter), there are techniques where malicious sites can impersonate logged-on users. Yes, there are complicated workarounds for the security problems (like double-submitting cookies). But if you’re starting out (like me), start off using JSON for public info you don’t mind having disclosed (like weather reports).

Callbacks

The JSON data may not be a raw object; it could be an object passed to a callback function, such as:

specialFunction( {"name" : "Bob", "email": "bob@example.com"} );

In this case, the script will run a function called specialFunction that you’ve defined, passing it an anonymous object with a “name” and “email” property. Your specialFunction can then do wild things with this information.

This technique is the same at heart – this is how Google originally passed your Gmail contacts. Sometimes you can specify the name of your callback function in the URL you use to access the JSON data – it depends on the data provider.

Real-life example: Currency conversion in InstaCalc

I recently added currency conversion to the InstaCalc Online Calculator, which was a good lesson in importing data:

• Find your data source. The Federal Bank of New York has an XML file with currency conversion data. That’s good enough for me!

• Convert data to JSON format. You can convert XML to JSON using XSLT transformations.

• Install XALAN so you can do XSL transformations using Java. A typical transformation looks like this:

java org.apache.xalan.xslt.Process -classpath "xalan.jar" 
-IN in.xml -XSL in.xslt > out.xml

Be careful for Java classpath issues, I had trouble and had to include the classpath to xalan.jar manually.

• Convert XML attributes to elements. Attributes can be difficult to access in the various XML to JSON scripts, so make them elements. Use this XSL to convert attributes to elements:

<?xml version="1.0" encoding="ISO-8859-1"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
   version="1.0">
  <xsl:output indent="yes" />
  <xsl:strip-space elements="*" />
  <xsl:template match="*">
    <xsl:copy>
      <xsl:if test="@*">
        <xsl:for-each select="@*">
          <xsl:element name="{name()}">
            <xsl:value-of select="." />
          </xsl:element>
        </xsl:for-each>
      </xsl:if>
      <xsl:apply-templates />
    </xsl:copy>
  </xsl:template>
</xsl:stylesheet>

Then process the XML file using XALAN:

java org.apache.xalan.xslt.Process -classpath "xalan.jar" 
-IN in.xml -XSL attributes-to-elements.xslt > no-attributes.xml

• Finally, convert the XML to JSON using this XSL transformation. There are several XML 2 JSON stylesheets out there if you search.

Run your XML through XALAN to create the final JSON data.

java org.apache.xalan.xslt.Process -classpath "xalan.jar" 
-IN no-attributes.xml -XSL xml2json.xslt > currencies.json

• Clean up your data. You have the raw JSON data as an object, but you may wish to wrap it in a function call. You can use a quick perl script to wrap “myFunction( )” around the data above so your file does a callback when executed.

Tada! Now you have JSON data ready to access using one of the techniques above (see the currency JSON data). Inside InstaCalc, I have a static reference to the file:

<script src="http://instacalc.com/data/currencies.json"><script>

Because the bank has some namespace information, I access variables like this inside the callback function: data["frbny:DataSet"];. Of course, the details of how you access your JSON data will change given the format of your XML and the exact stylesheet you used. Play around. If you look at the raw JSON file you can see the field names you need to use.

Bonus round: Updating the JSON. If your data changes often (like currency data does), you can put the above steps into a script and run it on a schedule. The next time your webpage is loaded, it will get the new JSON data.

Parting thoughts

JSON is a really easy way to exchange data – just think of it as including extra javascript files in your program. Read more at json.org, learn it, and love it. Good luck.