I’ve spent a good part of the day fielding calls from reporters about OpenCorporations.org and the changes that have led me to plan its demise — so far I’ve done phone interviews with The Guardian, the Globe and Mail and CBC Radio and I just finished taping a TV interview for Compass. Some points I’ve tried to make in those conversations:
- Search engines (like Google and Yahoo) index the pages on the Internet by having “robots” that visit all of the pages on the Internet: so Google can tell me where to go for pink ice cream cake because its robot visited this page and this page and this page — and 296,000 others — and found the words “pink ice cream cake” on them and added these words to its index.
- OpenCorporations.org has its own “search robot” that does exactly the same thing, albeit with the robot trained only on the Corporate Register pages.
- There was no robots.txt limitation on the Government of PEI webserver that prevented search engines from indexing the Corporate Register. There still isn’t. That’s where I took my “it’s okay if you index this content” cue.
- For years Google, and other search spiders, have been indexing PEI corporations data: here’s a Google search for ‘Homburg’ that shows this in action. So the changes not only shut off the tap for OpenCorporations, but also for the rest of the web.
- It’s completely within Government’s right to control the indexing of resources on their website and, even if it were possible, I wouldn’t try to circumvent the restrictions they’ve put in place, which clearly telegraph a “don’t index this” intent.
- Freedom of Information and Protection of Privacy legislation was not written to anticipate mashups like OpenCorporations: it’s an open question as to whether Government has a duty anticipate and act against potential remixing. I think the project was valuable if only for the focus it put on issues like this.
- I don’t think that Government acted with the aim of hiding anything, or preventing the lid from blowing off anything: I think they were forced to make an impromptu policy decision based on sudden focus on an unanticipated use of new technology; I happen to think they made the wrong policy decision, but I think their motives are pure.
If nothing else I’ve discovered through this experience that if you create a useful tool that’s especially useful for journalists, they will be especially interested if you have to shut it down. I happen to think that’s a good thing: and I’ve been generally impressed with the journalistic understanding of the subtleties of the story.