WikiGraph
MetaWiki-like service delivered via JavaScript. See 2012-12-01-BuildingWikiGraph
Short-term goals: Visible Backlinks and Twin Pages for this WikiLog.
Longer-term goals
- WikiWeb
- encourage use of Smashed Together Words (contra Free Link-s) for Automatic Linking in other WikiEngines
- 2016 update: WikiGraph With FreeLinks
- WikiGraphBrowser
Building with WebPy to run on my Linode.
Data in 3 SQL tables: spaces, pages, mentions
Scraped page lists by hand from a number of my Sister Sites
- My WikiLog has 15664 pages.
- Ignoring Free Link-s cases and other bad URL-s (like Smallest Federated Wiki) for now, as I'm the client.
For Visible Backlinks, started with 2003 file giving graph of my space (only 2804 pages at that time!) - good enough for starting to build.
Jan20'2013 - get static pieces working together
- edit
/MoinMoin/theme/__init__.page_info()
html += """\n<div id="backlinks"></div>
<script src="http://localhost:8081/backlinks.js"></script>
<noscript>Please enable JavaScript to view the BackLinks</noscript>"""
html += """\n<div id="twin_pages"></div>
<script src="http://localhost:8081/twinpages.js"></script>
<noscript>Please enable JavaScript to view the Twin Pages</noscript>"""
- routes
urls = (
'/twinpages.js','twinpages',
'/backlinks.js','backlinks',
- code
class backlinks:
def GET(self):
web.debug('calling backlinks')
resp = ''
backlinks = db.select('mentions', where="space_name = 'Web Seitz Wiki' AND page_mentioned = 'Bill Seitz'")
web.debug('backlinks: %s' % (backlinks))
if bool(backlinks):
resp = 'BackLinks: '
for backlink in backlinks:
resp += '<a href="%s%s">%s</a> | ' % ('http://webseitz.fluxent.com/wiki/', backlink.page_name, backlink.page_name)
resp = resp[:-3]
else:
resp = 'No backlinks!'
web.debug(len(resp))
return """
document.write('\
<div>\
<p>%s</p>\
</div>');
""" % (resp)
Feb22: clarify MVP plan
- focus on serving my WikiLog (and maybe Private Wiki)
- so no need to pass general params
- then move on to WikiGraphBrowser, since that shows the real power of WikiWord-s vs Free Link-s.
Feb26: buy ThirdPartyJavaScript book by Ben Vinegar and Anton Kovalyov (DisQus guys)
Feb27: get dynamic version of code working, using Http Referer and document.write()
Mar01: done styling content with inline div styles (how tacky).
Next: figure out how to parse all my current WikiLog pages for Visible Backlinks database.
- Mar10: done writing/testing locally
- Mar11: running against production.
- a few weird (MoinMoin) cases where
current
filename contents point at arevisions/
file that doesn't exist! Always a tuggle in these cases as to how hard to work at solving the general problem vs just hacking the exceptions and re-running. - have forward-refs for 15801 pages.
- a few weird (MoinMoin) cases where
Mar12: suck 96634 mentions into local db
- have some weird UTF8 issues, cleaned up file by hand
Next:
- convert SQLite to PostgreSQL format - see 2012-10-16-SettingUpHostingForSimplestThing
- get whole service deployed at Linode
Setting up hosting
Make a /srv/www/wikigraph.net/*
beside /srv/www/simplest-thing.com/*
- do chown/chmod bits.
- cp
/srv/.../app/start_uwsgi_app.sh
to parallel directory. Edit (port 9002).
Copy /etc/default/uwsgi
to /etc/default/uwsgi_wg
, edit the latter to set PYTHONPATH
to /srv/www...
.
Edit /opt/nginx/conf/nginx.conf
- add a header section for hostless-domain redirect and a header for the www.wikigraph.net host that points to the right place and uses port 9002 for uwsgi.
Copy WebPy code.py
to server.
Restart nginx. Kill/restart the uwsgi app for Simplest Thing. Start the uwsgi_wg app for WikiGraph.
Try to hit http://www.wikigraph.net/test - get "uWSGI Error - Python application not found"
- hit same point in setting up Simplest Thing, but had already handled tweaks from that learning here, so don't know what issue is...
Tweak start_uwsgi_app.sh
to call uwsgi_wg
instead of just uwsgi
. Kill/restart uwsgi.
Try to hit http://www.wikigraph.net/test - get 502 Bad Gateway. Realize didn't get the new uwsgi processes. Change the bash back. Get the "uWSGI Error" again.
Check the logs
- access.log - looks like status 500
- uwsgi_app.log - ah, notes that
import forms
failed (because not using that in this app yet, so hadn't copied it to server).
Tweak code.py, copy up, kill/restart uwsgi, hit page again. It does a redirect to /static/...
based on code copied from Simplest Thing, but I don't have that directory, so get 404 error.
Comment-out that check_authorized code, copy/restart/etc - now works!
Mar14: convert SQLite dump file to PostgreSQL
Mar20
- copy converted dump file to server
- create db
- import dump file; do some
COUNT
to confirm data in there - change laptop MoinMoin to point to production server JavaScript; reload page. Neither call returns data.
tail uwsgi_app.log
-ProgrammingError: permission denied for relation spaces
- just like in 2012-10-16-SettingUpHostingForSimplestThing- do the
GRANT
thing - now it works! - copy MoinMoin template file to Blue Host (after various file games).
- It's working! Now displaying Visible Backlinks and Twin Pages!
Apr15: tweak to let my local Private Wiki call it. Automatically works for creating-new page process so I can see if Twin Page already exists in my WikiLog and decide whether to add a Private page or stay public.
Future:
- have WikiLog "page does not exist yet" page include Twin Pages.
- WikiGraphBrowser!
- make way for backlinks-search to call db. Also change fulltext-search to call Google or maybe call WikiGraph.
- check scraper code for WikiWords with adjacent punctuation (like for italics)
- update lists
- support Free Link spaces (Phil Jones/SFW, Chris Dent/Tiddly Space...)
- 2016: WikiGraph With FreeLinks
Jul'2013 update: tweaked scraper code to catch some cases being missed before, re-ran
- 15963 pages
- 100451 mentions (6.3 mentions/page) (was 6.1 before - of course since both pieces changed it's hard to be sure...)
Twin Pages: pondering addition of Wikipedia
- 5M English articles, so don't feel like doing that
- 1000 "vital" articles, but suspect most wiki sites wouldn't have matches against very many of those (but maybe that's ok)
- could also do a hack, and always include a Wikipedia link that just generates a search...
- since Wikipedia is so unlike every other wiki, it actually sounds like a bad fit
Edited: | Tweet this! | Search Twitter for discussion