What do you like? Your ratings in multiple places – do you want to sync them?

The modern “what do you like?” systems are driving me nuts lately.  Every time I turn around, there’s a website offering to help me socialize, and what better way to find new friends (and new recommendations for media to consume) than by accumulating a bunch of ratings for the media I’ve already consumed?

Friends (and hopefully, to some degree) passers-by can then peruse my virtual “media shelves”, see what I’ve rated and how, and either (a) get or (b) give recommendations for stuff that’s related to what they/I have already seen/read/heard and liked.

Web 1.0: Amazon Recommendations

I’ve got > 1000 ratings for graphic novels accumulated on Amazon.com – which was a great way for years for me to get recommendations on other books by the same authors, but even more importantly and gratifyingly – to get recommendations on other books by authors I hadn’t yet read.  (I’ve always assumed some sort of Bayesian analysis that results in “people who liked your 4- and 5-star books also bought/owned/rated these items”.)

[I’d further tried to accumulate lists of ratings for music (for CDs I’ve purchased), but Amazon’s interface for this wasn’t nearly as sophisticated or predictive for music as it seems to be for books, so I’ve never gotten quite the same gratifying experience for music on Amazon.com.]

Web 1.0: Netflix Recommendations

I’ve got > 1000 ratings for movies accumulated on Netflix.com – which was a great way for years for me to get recommendations on other movies I’d like that have many of the same ephemeral qualities I enjoyed in the movies I rated most highly.  (Netflix has had a highly-publicized contest – which recently wrapped up – to come up with new ways of improving user recommendations, which to me meant that they’d exhausted all the available research into Bayesian and other mathematical analysis of the huge aggregations of data on what people liked, didn’t like, watched and marked “not interested”.)

Long before I was a Netflix subscriber though, I was (and am) a diehard advocate for IMDB.com.  [Hell, I was one of the lunatic adopters in the early days back when you had to submit queries to the IMDB via email.  Yeah, imagine browsing your favourite actor’s movies [and forget about TV – that didn’t count] by submitting a cryptically-formed email message and waiting the minutes it took for their servers to generate a response.  Fred Flintstone-style browsing.]  I’ve occasionally submitted a rating for movie through that site too, though I haven’t gotten any real benefit from it (except the knowledge that I’m helping to build the geek-slanted ratings that are the killer data set available from IMDB).

Web 2.0: social + recommendations

In the past year or so, I’ve fallen deeply in love with Facebook, Twitter and all the most interesting integrations with these “social platforms”.  With these platforms have come brand-new applications that allow you to rate movies/TV/books/music/whatever and not only get some kind of recommendations back from “the system”, but also to get much more specific and immediate feedback from those of your friends (or even “friends”) who’ve also signed up to use the application.  They see what you’ve rated, then respond with comments/replies/their own ratings, and can make much more specific (and personal, though statistically less predictive) suggestions of other stuff they want you to see/read/hear.

I love these – and while I’ve experimented with a bunch of these apps, I’ve gravitated to those apps that appear to have the greatest critical mass.  Not so much because I want a horde of strangers to help me find stuff, but because I’d like to reconnect with as many friends as possible and I hope they’re also at the apps I’ve picked.

So I’ve got ratings slowly accumulating at Goodreads.com, Flixster and a couple of others.  Goodreads has a great mobile site that makes it dead-easy to post a rating “on the go” with very few excess clicks, and the Flixster iPhone app is awesome *and* easy.  And there are dozens of other great sites where lots and lots of people are accumulating lots and lots of ratings data.

Problems: stale data, incomplete data, spread-too-thin efforts

After a while, I’ve noticed I’m spread thin across multiple places where these ratings are being accumulated.  It’s an unfortunate consequence of the abundance of such great sites and platforms, that I’m finding it hard to keep my ratings “in sync” between multiple places at once.  I have good intentions – and occasionally I’ll even follow through on those good intentions. 🙂

For example, I’ve got a ton of movie ratings in Netflix, but my primary interest for “sharing” movie ratings is moving to the Flixster app – mostly because it gives me a chance to get immediate feedback from a larger group of friends who catalogue their ratings and mini-reviews there via either Facebook or the iPhone app.  However, while I’m getting immediate gratification for my posts to Flixster, it’s not doing me any good in terms of system-generated (Bayesian) recommendations for other movies I want to watch. And when I go to my Netflix queue to add movies, I sometimes forget whether I’ve seen something (since I haven’t always rated those movies I’ve seen recently).

Similarly, the primary place I currently capture my ratings for graphic novels is in GoodReads (usually via the mobile-optimized web site that I access from my iPhone).  I’m not even getting any instant feedback from friends there, nor have I found any way to use the “crowd” of GoodReads users as a source for new recommendations.  However, there’s no alternative in my Web 1.0 world: Amazon doesn’t seem to have any way to add ratings on the go.  If you’re not going through their full browser, then you’re SOL.  (The Amazon iPhone app doesn’t do squat here, and neither does the mobile browser version – it’s almost as if they don’t care whether their customers like what they bought.)

I’m now split between worlds, and I suspect the world of Facebook/social media and other Web 3.0 apps will only make this worse – there’ll be more and more sites that all want you to provide some “sticky” information, that lures in more users ‘cause there’s a “crowd” there, and yet those ratings won’t be re-usable elsewhere.

This Ratings Data Ain’t Portable, My Friends

Yeah.  Twitter might have finally gotten religion that you “own” your Tweets, and Facebook *looks* like they’re convinced that you can and should have the ability to control your personal information/updates, but good luck trying to convince the thousands of little start-up apps out there, all hoping to lure you into their little walled garden and *keep* you there.

I expect that in 5-10 years, all these systems will be able to freely consume and re-use this data – the business world will have finally gotten over thinking this is their only “value add” (and will have found some even more sexy way to separate you from your money).

However, for the forseeable future, these multiple ratings systems will continue to live as non-interoperable data islands.  That means heavy “data generators” like me will have to make some pretty dopey (and unavoidable) decisions:

  1. When you find a new, even-more-attractive place to catalogue your consumption and how much you enjoyed it, do you abandon all the invested effort you put into the last one?
  2. If you don’t want to abandon all that “legacy data”, how will you migrate it to the new system? Just devote a freakin’ weekend to the prospect of clicking like a spastic lab rat, replicating each rating from one system to the other?  Or do you go even further down the rabbit hole and learn how to export the data (if that’s supported) from one and import to another – or go completely over the cliff edge into writing yourself some web-scraping scripts that pull the data by force out of systems that don’t have a supported import/export interface (API)?
  3. And if you actually *want* to maintain a presence in more than one system – e.g. if you find some ongoing benefit in having current presence in both Netflix and Flixster?  Well gods help you then – you’re screwed into a life of regular repeated self-inflicted punishment.

I’ve really lost my mind – thinking there’s got to be some way to actually pull off (3) without feeling like a character in a Kafka novel (and no, I’ve never read Kafka, so don’t crucify me for a misspoken cultural reference).

How’s About a Ratings Sync App?

Yeah, why the hell not?  Why not just burn the next years’ worth of weekends writing an extensible framework for us to be able to download, manipulate, upload and synchronize (i.e. manage and resolve the inevitable conflicts) the ratings data?  Isn’t is just like me, to think of doing some thankless job like this, in the hopes that some morsel of thanks comes through from some other hapless geek like me?

Sure, what the hell.

In fact, I’ve invested a whole bunch of time into this harebrained notion already.  Yes, I’ve written myself a bunch of code that attempts to provide an extensible, pluggable framework in which multiple “ratings” providers could be wired in, and between which synchronization could occur.

I actually dream that one day, users like me could:

  • Fire up this app
  • Connect to one of their ratings aggregating web sites
  • Download all ratings for whatever “things” are rated on that site
  • Select another compatible ratings web site (e.g. another books-rating web site if you just download book ratings)
  • Configure a translation between the two web sites (e.g. one site rates 1-5, the other site rates 1-10, so map 1 = 2, 2 = 4, 3 = 6, 4 = 8, 5 = 10; or if you’d prefer, 1 = 1, 2 = 3, 3 = 5, 4 = 7, 5 = 9)
  • Upload the translated ratings to the second site, thus synchronizing your ratings from one site to the next

This’ll require mapping out the APIs for each ratings-aggregating site, implementing an incredible flexible and robust local schema for the data, and figuring out all the different ways that different sites identify what to the human mind is an easy-to-identify product.

[And to think, this whole idea came about as a way to figure out how to migrate the ratings from my old Netflix account (to which I’ve still got access, but only just barely – due to the grace of an old housemate) to a new one that *I* own, and from which I could actually do Netflix Instant Watch in my own personality (and with my own IW queue).  I’m seriously considering just paying them for their subscription for the rest of my life, so I never have to lose those 2956 movie ratings.]


Anyone out there got a better idea?

Anyone else crazy enough to want to help out with this?

Anyone out there want to see this app see the light of day?

InfoPath forms error – one of the most obscure I’ve ever seen…

I had an InfoPath web form configured, working well, and ready to deploy.  I let my colleague know about it and when he tried it out a day or two later, he was getting an error message I hadn’t myself seen:


There has been an error while processing the form.

Click Continue to resume filling out the form. You may want to check your form data for errors.

Click Start Over to load a new copy of the form.

Show error details

When I browsed the Details of the error message, InfoPath Forms Services showed me this:

An error occurred accessing a data source.

An entry has been added to the Windows event log of the server.
Log ID: 5566

Not much help there, but I kept trying to figure this out.  I tried from different browsers (same error) and from different PCs (still same).

I got the bright idea to check the settings in the Data Connection in the InfoPath template I’d designed, but everything looked the same – even the URL that I was using to retrieve an XML representation of the List data (according to this helpful blog entry on “Accessing SharePoint List Data as XML” and “Populating form data from SharePoint List Views”).  However, when I tried completing the Data Connection wizard for this existing connection, I was getting blasted with this error:

The file is not a valid XML file.
The file is not a valid XML document.
A document must contain exactly one root element.
  Line 1, Position 0

I wondered if I’d messed up the permissions on the List, so I verified that I and my colleague had Read & Contribute permissions (so the Form would load), and I even added back the “System Account” with its Limited Access (using the “trick” documented in this blog article).  Still nothing.

When I tried to load that URL in a browser, the page it returned was blank, and even when I looked at the View Source, there wasn’t anything helpful there:

I retried this over and over for a few days, re-examining the URL in the Data Connection, triple-checking the GUIDs (thinking they must’ve gotten changed somehow during a recent tweak to the List), and wondering what I’d done wrong (or what had gotten horked up on the servers).

And the Solution Is…

Eventually I tried some googling on “A document must contain exactly one root element”, and finally returned to my old friends, the InfoPathDev.com and SharePointBlogs.com sites.  When I arrived back at the “Accessing SharePoint List Data as XML” article, one of the comments jumped out at me:

I can’t get this to work on a List that contains a Lookup field.  I just get a blank page in the browser. Have you encountered this too? Any ideas?

Bingo!  One of the recent tweaks I’d made was to convert a field in the list from using the “Choice” data type to the “Lookup” data type.  (Note: using a Lookup to a separate List enables me to delegate maintenance of that list of Choices to my team members, without asking them to muddle around in the bowels of my List’s Column settings.)

All I had to do was to remove the one Lookup field from the View I was retrieving in this Data Connection, and the web Form started working again (without any changes or re-Publishing necessary, even!).  I even confirmed that the URL, when used in a browser, would return a readable XML document (though you have to View Source on the returned page, which looks just as blank in IE8 as the failing one).

Lesson: if you’re going to leverage SharePoint Lists as XML data sources in your InfoPath Forms, make sure that the View you select does NOT include a Lookup field.  I told you this was obscure…

Categories XML

Free XML tools? Trying to find a WYSIWYG editor for XML, XSLT – how hard can it be?

I’ve been using Microsoft’s Threat Analysis and Modeling tool (more on my experiences later), and one of the things I’ve determined is that I need an XML/XSLT editing/authoring tool to help me wade through all the information that’s buried in the threat model documents it generates (which is all written in well-formatted XML).

I’ve spent a few weeks trying desperately to find one tool that would allow me to do the following:

  • Get a quick overview of the XML without exposing me to all the raw code
  • Explore the hierarchy of XML in a treeview
  • Allow me to automatically collapse or hide certain elements and branches of the XML so that I can skip e.g. the GUID element in every object
  • Allow me to drag & drop XML elements around the document (or at least easy-to-use cut and paste)
  • Allow me to browse a self-describing set of the XML tags that are available to use in this document e.g. explore the document’s schema in tree form; collate a list of the XSL tags already in use in the document (not just all those potential tags supported in XSL 1.0 or 2.0)

A few abandonware tools that are mentioned all over the ‘Net, but which are no longer options:

  • ActiveState Visual XSLT: ActiveState delisted this years ago, and I cannot find a copy anywhere on the ‘Net for the life of me
  • Altova XMLSpy 2004 Home Edition: still available for download from Altova’s software archive, but they have removed all traces of the License keys that could be obtained for free
  • IBM XSL Editor: now part of some Websphere server-side engine

Some freeware apps I’ve found:

  • XMLmind Personal Edition: java-based GUI editor.  Provides collapsible tree view of document contents, inline editor for element text (hiding all the nasty tag content and code), and Provides downloadable add-ons to enable editing documents based on many different DTDs.
  • Microsoft XML Notepad 2007: horrible UI, but at least it abstracts away some of the complexity of XML…
  • Notepad++: does a good job of highlighting XML syntax and allowing you to collapse element branches, but doesn’t help my primary problem, which is…
  • XML Cooktop: XML syntax highlighting, processing XML with XSLTs and viewing the results
  • Butterfly XML: decent hierarchical view (and in my personal opinion, it beats out XML Notepad 2007), doesn’t scare off the user with tagged code, but no cut & paste capability (just add or delete).
  • Xerlin: Java-based XML editor
  • Vex: a “visual” XML editor 
  • Jaxe
  • XML Workbench

Visual Studio 2005 has some XML capabilities, but strangely nowhere near the level of “friendliness” that I’ve come to expect from this IDE:

  • It’ll open XML & XSLT just fine, perform the usual syntax highlighting, and even autocomplete any code that I try to add.
  • However, it just shows me the “raw code” – I can’t find any way to “hide” the code and just browse through this file in a more compact, abstract, more user-friendly way.
  • You know how you can switch between Design Mode and Code in VS2005 (e.g. creating a WinForms app)?  Or how FrontPage used to allow you to do the same thing?  That’s what I want.
  • Then I can learn some of the basics of what the code is doing by association, but in the meantime I can “get something done” without having to spend 15-20 hours learning about XML & XSLT syntax, language operators and all this minutia.
  • It’d be really cool if there was some sort of Object Browser tree view of the XSD (XML schema) or a “Toolbox” like collection of the kinds of code objects that I could add (that wouldn’t scare me off as much as raw code in unknown languages does).
  • Finally, I haven’t found any freebie XML-editing add-ons for VS2005.  [Other than the Visual XSLT from ActiveState that appears to have disappeared from existence.]
  • The next version of Visual Studio (“Orcas”) promises some improvements in dealing with XML & XSLT, but nothing earth shattering (at least not for my needs)

Actually, one of the best tools I’ve tried is another commercial app that I would never have expected to make “browsing and editing XML” as visual and flexible as it does: MindManager.

  • MindManager’s native document format is a compound XML document, and it’s able to open and save in unmolested XML format.
  • If you open an XML document, it’ll treat elements as Topics and element text as Callouts, making it easy to cut & paste or drag & drop elements from one part of the hierarchy (which looks like a tree with branches) to another.
  • It’s very slick, looks very simple but has all sorts of extensible power behind it.

And then there’s a raft of commercial apps that might do the trick, but which cost way more $$ than I have budget for:

  • Altova XMLSpy: For my purposes, this is no better than Visual Studio (even though it’s *loaded* with lots of additional features that I’m sure every XML goon would love).
  • Altova StyleVision: wow, how *un*suitable is that?  It doesn’t even open XSLT files – it only uses its own proprietary format, and while it seems to have lots of different ways to look at the content of the style sheet you’re working on, none of it seemed intuitive or familiar.  Probably makes more sense to someone who’s been using Altova tools for a while.
  • Stylus Studio XML: I like the visual XSLT mapper, the XSLT “backmapper”, their Sense:X feature, their ability to drag-and-drop from source XML to the XSLT (with intelligent handling of the resulting object), and their WYSIWYG XSLT designer (“You Design the HTML; Stylus Studio Writes the XSLT”)!!  Man, with all this going for it, I’m going to at *least* spend a few days with the eval version. 🙂


Surprised?  I was.  MindManager and Stylus Studio XML were *not* on the short list – they’ve (a) never been mentioned by anyone I talked to who had any familiarity with XML, and (b) if you’d asked me about any tools that have XML editing capabilities I’d never have been able to come up with these.


Sidebar: Lemme just mea culpa for a sec – it’s not that I can’t code (I taught myself VB.NET from scratch), but it seems horribly inefficient to try to edit existing XSLT documents only after I have to learn every little aspect of XSLT and XML syntax.  Doesn’t it?