Zwiki Freebsd Stability Problems

It turns out there are problems with the configuration of Python for FreeBSD, which tends to result in instability, in particular in Zwiki installations (which use lots of Regular Expression parsing). Here's my log. (For reference will track some Unix Commands.)

Context: going from...

  • going from zope 2.3.2 (python 1.5.2) w/ zwiki 0.9.5

  • to zope 2.5.1 (python 2.1.3) w/ zwiki 0.17

June 5, 2003

Got things "up", but consistently getting Proxy Error 502 when trying to view Front Page. A "normal" page is still immediately viewable, so I don't think zope is actually crashed.

  • Lots of pages that don't make much sense to me yet.

  • Quick hack: I noticed my target cache size was 400, even though I had 3200+ objects across caches. (And 15,727 objects in the database.) So I upped the target to 3300. Immediately the number in the cache turned to 5334. One Simon page notes that you can get 2+ threads, and the cache will be multiplied by the threads. That didn't make the page viewable.

  • Tried changing Front Page to just show 20 items instead of 50, but can't even save a change.

  • Just went to Control Panel to try restarting service, notice that it says it's only been up for 1 min! So maybe it is crashing all the time, but it restarts fast.

    • yes - I try for Front Page, it waits 10 seconds, give me the Proxy Error, then I immediately look at Control Panel and see that the server's only been up for 5 seconds.
  • to reduce size of Front Page, take local version hack to only show 20, then export/copy/import. That seems to be working for now.

    • a bit later, it died again
  • also run a Cache Full Sweep, just for the heck of it.

    • looked 15 minutes later, see 3250 items in log, so it's stuck with a single thread
  • look at /usr/local/zope/instance/var/events.log (tail -1000 events.log)

  • my system eventually was so locked I couldn't get into anything. I called iMeme and they whacked it (the emergency zope restart button on the support pages should do the trick next time).

    • I tried talking to them at IRC, but nobody was listening, though some were logged on: irc.imeme.net /join #imeme

    • tried to kill using: killall -9 python

      • Then start your Zope by running

      • /usr/local/zope/instance/storage_start

      • /usr/local/zope/instance/zope_start

    • I removed the My Blog Roll from Front Page to reduce the regexp load. Then shortened My Recent Key Pages also.

      • this probably isn't sufficient, since I just checked the Control Panel and saw that the server just restarted again.
    • Jon @ iMeme says he's already made the FreeBSD python patch! So I posted at zwiki.org and emailed Simon.

    • status

      • at 6:21pm, it's been up 1h11m

      • at 10pm, it's only been up 18m, so it rebooted at least once

    • hmm, I wonder whether changing the page type would help? Of course, can't do it through the ZMI because of the Zope GetSlice Bug!

      • hmm, something broke the rendering. Will have to look at this later, including the pageTypes of more static wiki pages.

Jun7

  • looks like Jon's patch doesn't include the larger stack size

  • <http://zwiki.org/Patch Python 213>

  • Jon says You can cvsup the ports tree and just have it build the version of python you need from that its the same setup we use though I'm not sure the date on the python version I installed. The only stack increase I did was the one offered on the Plone site.

  • not sure what I'm doing, but I'll try to get at least partway

  • look in /usr/ports/ - there's no lang/

    • try to create with Win S C P, no permissions

    • try to do via ssh - have to change to su

  • download file to laptop, copy to imeme - have to do chown

  • do tar -x

    • says can't open /dev/sa0
  • try unzipping on my machine, won't let me because of a funny file name (at least when I do it auto)

    • should probably rename the file, copy to imeme, then rename again

      • argh, can't rename file within Win Zip!
  • notice that files have full root set, so maybe need to do untar from root directory.

  • after gnashing, realize need to use 'gunzip' on a .tgz file - that works, generating a .tar file

  • but "tar x usr-....*.tar" still doesn't work - gives "not in archive" message.

    • need to use 'f' modifier to specify tar file name, so 'tar xvf [filename]' - that works!

    • some confusion: some of document says '/usr/ports/langs...' others say /usr/ports/lang...' - the file uses the singular, so it just re-created some semi-parallel directories

  • do 'make extract', it says "bad value or PYTHON_VERSION: python2.1. Use one of python1.5, python1.6 or python2.0 (default)... Error code 1

  • Jon says I needed a new CVS base directory, which he gave me. Now I can type : make all - then : kill off all running pythons (killall -9 python) - then : make install

    • 'make all'

    • download .h file , change <<17 to <<20, upload (to work/Python-2.1.3/Python/) (chown again)

    • then killall, then make-install

    • then storage_startup, then zope_startup

    • up and running at 10:45pm

    • still up at 7am!!!

Jun8

stability looking good

whoops, nope, looks like it rebooted at 1:30pm.


Want to download log file and check changes in frequency of reboot. (note times are 2 hours ahead of EST) ::

2003-06-05T05:44:11 ERROR(200) zdaemon zdaemon: Thu Jun 5 05:44:11 2003: Aiieee! 56733 exited with error code: 11 2003-06-05T05:44:32 ERROR(200) zdaemon zdaemon: Thu Jun 5 05:44:32 2003: Aiieee! 68608 exited with error code: 11 2003-06-05T05:45:25 ERROR(200) zdaemon zdaemon: Thu Jun 5 05:45:25 2003: Aiieee! 68656 exited with error code: 11 2003-06-05T05:46:37 ERROR(200) zdaemon zdaemon: Thu Jun 5 05:46:37 2003: Aiieee! 69036 exited with error code: 11 2003-06-05T05:59:25 ERROR(200) zdaemon zdaemon: Thu Jun 5 05:59:25 2003: Aiieee! 69210 exited with error code: 11 2003-06-05T06:15:11 ERROR(200) zdaemon zdaemon: Thu Jun 5 06:15:11 2003: Aiieee! 72207 exited with error code: 11 2003-06-05T06:15:26 ERROR(200) zdaemon zdaemon: Thu Jun 5 06:15:26 2003: Aiieee! 76591 exited with error code: 11 2003-06-05T07:00:31 ERROR(200) zdaemon zdaemon: Thu Jun 5 07:00:31 2003: Aiieee! 76622 exited with error code: 11 2003-06-05T07:15:26 ERROR(200) zdaemon zdaemon: Thu Jun 5 07:15:26 2003: Aiieee! 87731 exited with error code: 11 2003-06-05T07:22:13 ERROR(200) zdaemon zdaemon: Thu Jun 5 07:22:13 2003: Aiieee! 91673 exited with error code: 11 2003-06-05T07:29:25 ERROR(200) zdaemon zdaemon: Thu Jun 5 07:29:25 2003: Aiieee! 93250 exited with error code: 11 2003-06-05T07:29:46 ERROR(200) zdaemon zdaemon: Thu Jun 5 07:29:46 2003: Aiieee! 94866 exited with error code: 11 2003-06-05T07:43:58 ERROR(200) zdaemon zdaemon: Thu Jun 5 07:43:58 2003: Aiieee! 94909 exited with error code: 11 2003-06-05T07:44:27 ERROR(200) zdaemon zdaemon: Thu Jun 5 07:44:27 2003: Aiieee! 98524 exited with error code: 11 2003-06-05T07:44:56 ERROR(200) zdaemon zdaemon: Thu Jun 5 07:44:56 2003: Aiieee! 98609 exited with error code: 11 2003-06-05T07:45:43 ERROR(200) zdaemon zdaemon: Thu Jun 5 07:45:43 2003: Aiieee! 98662 exited with error code: 11 2003-06-05T07:46:33 ERROR(200) zdaemon zdaemon: Thu Jun 5 07:46:33 2003: Aiieee! 99034 exited with error code: 11 2003-06-05T07:58:45 ERROR(200) zdaemon zdaemon: Thu Jun 5 07:58:45 2003: Aiieee! 99160 exited with error code: 11 2003-06-05T08:12:47 ERROR(200) zdaemon zdaemon: Thu Jun 5 08:12:47 2003: Aiieee! 2080 exited with error code: 11 2003-06-05T08:15:18 ERROR(200) zdaemon zdaemon: Thu Jun 5 08:15:18 2003: Aiieee! 6464 exited with error code: 11 2003-06-05T09:15:25 ERROR(200) zdaemon zdaemon: Thu Jun 5 09:15:25 2003: Aiieee! 6850 exited with error code: 11 2003-06-05T09:28:24 ERROR(200) zdaemon zdaemon: Thu Jun 5 09:28:24 2003: Aiieee! 27340 exited with error code: 11 2003-06-05T09:28:49 ERROR(200) zdaemon zdaemon: Thu Jun 5 09:28:49 2003: Aiieee! 30404 exited with error code: 11 2003-06-05T09:32:15 ERROR(200) zdaemon zdaemon: Thu Jun 5 09:32:15 2003: Aiieee! 30455 exited with error code: 11 2003-06-05T09:32:33 ERROR(200) zdaemon zdaemon: Thu Jun 5 09:32:33 2003: Aiieee! 31515 exited with error code: 11 2003-06-05T09:37:13 ERROR(200) zdaemon zdaemon: Thu Jun 5 09:37:13 2003: Aiieee! 32560 exited with error code: 11 2003-06-05T14:02:33 ERROR(200) zdaemon zdaemon: Thu Jun 5 14:02:33 2003: Aiieee! 32634 exited with error code: 11 2003-06-05T14:15:28 ERROR(200) zdaemon zdaemon: Thu Jun 5 14:15:28 2003: Aiieee! 98995 exited with error code: 11 2003-06-05T15:08:59 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:08:59 2003: Aiieee! 2418 exited with error code: 11 2003-06-05T15:09:16 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:09:16 2003: Aiieee! 17020 exited with error code: 11 2003-06-05T15:09:31 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:09:31 2003: Aiieee! 17103 exited with error code: 11 2003-06-05T15:15:25 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:15:25 2003: Aiieee! 17134 exited with error code: 11 2003-06-05T15:23:08 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:23:08 2003: Aiieee! 18767 exited with error code: 11 2003-06-05T15:28:06 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:28:06 2003: Aiieee! 20467 exited with error code: 11 2003-06-05T15:31:11 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:31:11 2003: Aiieee! 21576 exited with error code: 11 2003-06-05T15:35:46 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:35:46 2003: Aiieee! 22553 exited with error code: 11 2003-06-05T15:36:13 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:36:13 2003: Aiieee! 23557 exited with error code: 11 2003-06-05T15:44:13 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:44:13 2003: Aiieee! 23658 exited with error code: 11 2003-06-05T15:49:04 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:49:04 2003: Aiieee! 25583 exited with error code: 11 2003-06-05T15:59:50 ERROR(200) zdaemon zdaemon: Thu Jun 5 15:59:50 2003: Aiieee! 28591 exited with error code: 11 2003-06-05T16:03:15 ERROR(200) zdaemon zdaemon: Thu Jun 5 16:03:15 2003: Aiieee! 29260 exited with error code: 11 2003-06-05T16:07:56 ERROR(200) zdaemon zdaemon: Thu Jun 5 16:07:56 2003: Aiieee! 30366 exited with error code: 11 2003-06-05T16:11:13 ERROR(200) zdaemon zdaemon: Thu Jun 5 16:11:13 2003: Aiieee! 31384 exited with error code: 11 2003-06-05T16:25:10 ERROR(200) zdaemon zdaemon: Thu Jun 5 16:25:10 2003: Aiieee! 32383 exited with error code: 11 2003-06-05T16:35:06 ERROR(200) zdaemon zdaemon: Thu Jun 5 16:35:06 2003: Aiieee! 36154 exited with error code: 11 2003-06-05T16:36:41 ERROR(200) zdaemon zdaemon: Thu Jun 5 16:36:41 2003: Aiieee! 38170 exited with error code: 11 2003-06-05T16:37:29 ERROR(200) zdaemon zdaemon: Thu Jun 5 16:37:29 2003: Aiieee! 38404 exited with error code: 11 2003-06-05T16:42:35 ERROR(200) zdaemon zdaemon: Thu Jun 5 16:42:35 2003: Aiieee! 38527 exited with error code: 11 2003-06-05T16:51:24 ERROR(200) zdaemon zdaemon: Thu Jun 5 16:51:24 2003: Aiieee! 39833 exited with error code: 11 2003-06-05T19:04:53 ERROR(200) zdaemon zdaemon: Thu Jun 5 19:04:53 2003: Aiieee! 42057 exited with error code: 11 2003-06-05T19:06:18 ERROR(200) zdaemon zdaemon: Thu Jun 5 19:06:18 2003: Aiieee! 75618 exited with error code: 11 2003-06-05T19:06:51 ERROR(200) zdaemon zdaemon: Thu Jun 5 19:06:51 2003: Aiieee! 76006 exited with error code: 11 2003-06-05T19:20:55 ERROR(200) zdaemon zdaemon: Thu Jun 5 19:20:55 2003: Aiieee! 76068 exited with error code: 11 2003-06-05T19:22:02 ERROR(200) zdaemon zdaemon: Thu Jun 5 19:22:02 2003: Aiieee! 79687 exited with error code: 11 2003-06-05T19:22:19 ERROR(200) zdaemon zdaemon: Thu Jun 5 19:22:19 2003: Aiieee! 79922 exited with error code: 11 2003-06-05T19:22:44 ERROR(200) zdaemon zdaemon: Thu Jun 5 19:22:44 2003: Aiieee! 79954 exited with error code: 11 2003-06-05T20:21:38 ERROR(200) zdaemon zdaemon: Thu Jun 5 20:21:38 2003: Aiieee! 94768 exited with error code: 11 2003-06-05T20:22:57 ERROR(200) zdaemon zdaemon: Thu Jun 5 20:22:57 2003: Aiieee! 94942 exited with error code: 11 2003-06-05T20:23:32 ERROR(200) zdaemon zdaemon: Thu Jun 5 20:23:32 2003: Aiieee! 95121 exited with error code: 11 2003-06-05T20:57:45 ERROR(200) zdaemon zdaemon: Thu Jun 5 20:57:45 2003: Aiieee! 95217 exited with error code: 11 2003-06-05T20:59:56 ERROR(200) zdaemon zdaemon: Thu Jun 5 20:59:56 2003: Aiieee! 4085 exited with error code: 11 2003-06-05T21:09:35 ERROR(200) zdaemon zdaemon: Thu Jun 5 21:09:35 2003: Aiieee! 5172 exited with error code: 11 2003-06-06T01:58:48 ERROR(200) zdaemon zdaemon: Fri Jun 6 01:58:48 2003: Aiieee! 34782 exited with error code: 11 2003-06-06T01:59:23 ERROR(200) zdaemon zdaemon: Fri Jun 6 01:59:23 2003: Aiieee! 86989 exited with error code: 11 2003-06-06T03:20:09 ERROR(200) zdaemon zdaemon: Fri Jun 6 03:20:09 2003: Aiieee! 87740 exited with error code: 11 2003-06-06T03:36:14 ERROR(200) zdaemon zdaemon: Fri Jun 6 03:36:14 2003: Aiieee! 28285 exited with error code: 11 2003-06-06T10:36:38 ERROR(200) zdaemon zdaemon: Fri Jun 6 10:36:38 2003: Aiieee! 84711 exited with error code: 11 2003-06-06T13:26:37 ERROR(200) zdaemon zdaemon: Fri Jun 6 13:26:37 2003: Aiieee! 97747 exited with error code: 11 2003-06-06T15:55:37 ERROR(200) zdaemon zdaemon: Fri Jun 6 15:55:37 2003: Aiieee! 99765 exited with error code: 11 2003-06-06T16:43:04 ERROR(200) zdaemon zdaemon: Fri Jun 6 16:43:04 2003: Aiieee! 77825 exited with error code: 11 2003-06-06T17:01:59 ERROR(200) zdaemon zdaemon: Fri Jun 6 17:01:59 2003: Aiieee! 19829 exited with error code: 11 2003-06-06T17:03:03 ERROR(200) zdaemon zdaemon: Fri Jun 6 17:03:03 2003: Aiieee! 66566 exited with error code: 11 2003-06-06T20:38:53 ERROR(200) zdaemon zdaemon: Fri Jun 6 20:38:53 2003: Aiieee! 68304 exited with error code: 11 2003-06-06T21:45:13 ERROR(200) zdaemon zdaemon: Fri Jun 6 21:45:13 2003: Aiieee! 8498 exited with error code: 11 2003-06-06T22:07:15 ERROR(200) zdaemon zdaemon: Fri Jun 6 22:07:15 2003: Aiieee! 2081 exited with error code: 11 2003-06-06T22:08:08 ERROR(200) zdaemon zdaemon: Fri Jun 6 22:08:08 2003: Aiieee! 65099 exited with error code: 11 2003-06-06T22:11:59 ERROR(200) zdaemon zdaemon: Fri Jun 6 22:11:59 2003: Aiieee! 69105 exited with error code: 11 2003-06-07T02:27:50 ERROR(200) zdaemon zdaemon: Sat Jun 7 02:27:50 2003: Aiieee! 77106 exited with error code: 11 2003-06-07T02:29:29 ERROR(200) zdaemon zdaemon: Sat Jun 7 02:29:29 2003: Aiieee! 40743 exited with error code: 11 2003-06-07T05:52:57 ERROR(200) zdaemon zdaemon: Sat Jun 7 05:52:57 2003: Aiieee! 44512 exited with error code: 11 2003-06-07T15:30:17 ERROR(200) zdaemon zdaemon: Sat Jun 7 15:30:17 2003: Aiieee! 19608 exited with error code: 11 2003-06-07T17:38:38 ERROR(200) zdaemon zdaemon: Sat Jun 7 17:38:38 2003: Aiieee! 35806 exited with error code: 11 2003-06-07T17:39:06 ERROR(200) zdaemon zdaemon: Sat Jun 7 17:39:06 2003: Aiieee! 8202 exited with error code: 11 2003-06-07T18:08:28 ERROR(200) zdaemon zdaemon: Sat Jun 7 18:08:28 2003: Aiieee! 8497 exited with error code: 11 2003-06-07T23:20:11 ERROR(200) zdaemon zdaemon: Sat Jun 7 23:20:11 2003: Aiieee! 88073 exited with error code: 11 2003-06-07T23:24:19 ERROR(200) zdaemon zdaemon: Sat Jun 7 23:24:19 2003: Aiieee! 74714 exited with error code: 11 2003-06-08T00:47:33 ERROR(200) zdaemon zdaemon: Sun Jun 8 00:47:33 2003: Aiieee! 86720 exited with error code: 11 2003-06-09T00:45:41 ERROR(200) zdaemon zdaemon: Mon Jun 9 00:45:41 2003: Aiieee! 18785 exited with error code: 11 2003-06-09T11:28:05 ERROR(200) zdaemon zdaemon: Mon Jun 9 11:28:05 2003: Aiieee! 63639 exited with error code: 11 2003-06-09T11:31:36 ERROR(200) zdaemon zdaemon: Mon Jun 9 11:31:36 2003: Aiieee! 59055 exited with error code: 11 2003-06-09T11:32:52 ERROR(200) zdaemon zdaemon: Mon Jun 9 11:32:52 2003: Aiieee! 68023 exited with error code: 11 2003-06-09T12:34:42 ERROR(200) zdaemon zdaemon: Mon Jun 9 12:34:42 2003: Aiieee! 72762 exited with error code: 11 2003-06-09T16:40:23 ERROR(200) zdaemon zdaemon: Mon Jun 9 16:40:23 2003: Aiieee! 13875 exited with error code: 11 2003-06-10T07:39:44 ERROR(200) zdaemon zdaemon: Tue Jun 10 07:39:44 2003: Aiieee! 11385 exited with error code: 11 2003-06-10T12:42:58 ERROR(200) zdaemon zdaemon: Tue Jun 10 12:42:58 2003: Aiieee! 21168 exited with error code: 11 2003-06-10T16:50:56 ERROR(200) zdaemon zdaemon: Tue Jun 10 16:50:56 2003: Aiieee! 34937 exited with error code: 11 2003-06-10T18:07:47 ERROR(200) zdaemon zdaemon: Tue Jun 10 18:07:47 2003: Aiieee! 28985 exited with error code: 11 2003-06-10T18:15:26 ERROR(200) zdaemon zdaemon: Tue Jun 10 18:15:26 2003: Aiieee! 37789 exited with error code: 11 2003-06-10T18:46:20 ERROR(200) zdaemon zdaemon: Tue Jun 10 18:46:20 2003: Aiieee! 48744 exited with error code: 11 2003-06-10T18:48:11 ERROR(200) zdaemon zdaemon: Tue Jun 10 18:48:11 2003: Aiieee! 92211 exited with error code: 11 2003-06-10T18:51:44 ERROR(200) zdaemon zdaemon: Tue Jun 10 18:51:44 2003: Aiieee! 94985 exited with error code: 11 2003-06-10T22:33:09 ERROR(200) zdaemon zdaemon: Tue Jun 10 22:33:09 2003: Aiieee! 2346 exited with error code: 11 2003-06-10T23:09:13 ERROR(200) zdaemon zdaemon: Tue Jun 10 23:09:13 2003: Aiieee! 66535 exited with error code: 11 2003-06-10T23:54:58 ERROR(200) zdaemon zdaemon: Tue Jun 10 23:54:58 2003: Aiieee! 30495 exited with error code: 11 2003-06-10T23:55:28 ERROR(200) zdaemon zdaemon: Tue Jun 10 23:55:28 2003: Aiieee! 6541 exited with error code: 11 2003-06-10T23:59:50 ERROR(200) zdaemon zdaemon: Tue Jun 10 23:59:50 2003: Aiieee! 6955 exited with error code: 11 2003-06-11T00:07:55 ERROR(200) zdaemon zdaemon: Wed Jun 11 00:07:55 2003: Aiieee! 20424 exited with error code: 11 2003-06-11T00:09:29 ERROR(200) zdaemon zdaemon: Wed Jun 11 00:09:29 2003: Aiieee! 43123 exited with error code: 11 2003-06-11T00:15:14 ERROR(200) zdaemon zdaemon: Wed Jun 11 00:15:14 2003: Aiieee! 46657 exited with error code: 11 2003-06-11T01:20:54 ERROR(200) zdaemon zdaemon: Wed Jun 11 01:20:54 2003: Aiieee! 47907 exited with error code: 11 2003-06-11T01:21:29 ERROR(200) zdaemon zdaemon: Wed Jun 11 01:21:29 2003: Aiieee! 53993 exited with error code: 11 2003-06-11T01:28:52 ERROR(200) zdaemon zdaemon: Wed Jun 11 01:28:52 2003: Aiieee! 54397 exited with error code: 11 2003-06-11T01:34:43 ERROR(200) zdaemon zdaemon: Wed Jun 11 01:34:43 2003: Aiieee! 77124 exited with error code: 11 2003-06-11T01:46:26 ERROR(200) zdaemon zdaemon: Wed Jun 11 01:46:26 2003: Aiieee! 94865 exited with error code: 11 2003-06-11T02:38:07 ERROR(200) zdaemon zdaemon: Wed Jun 11 02:38:07 2003: Aiieee! 30051 exited with error code: 11 2003-06-11T02:51:42 ERROR(200) zdaemon zdaemon: Wed Jun 11 02:51:42 2003: Aiieee! 76694 exited with error code: 11 2003-06-11T02:52:41 ERROR(200) zdaemon zdaemon: Wed Jun 11 02:52:41 2003: Aiieee! 11463 exited with error code: 11 2003-06-11T03:09:58 ERROR(200) zdaemon zdaemon: Wed Jun 11 03:09:58 2003: Aiieee! 12908 exited with error code: 11 2003-06-11T09:17:52 ERROR(200) zdaemon zdaemon: Wed Jun 11 09:17:52 2003: Aiieee! 84378 exited with error code: 11 2003-06-11T09:44:50 ERROR(200) zdaemon zdaemon: Wed Jun 11 09:44:50 2003: Aiieee! 2721 exited with error code: 11 2003-06-11T10:15:16 ERROR(200) zdaemon zdaemon: Wed Jun 11 10:15:16 2003: Aiieee! 48833 exited with error code: 11 2003-06-11T11:40:37 ERROR(200) zdaemon zdaemon: Wed Jun 11 11:40:37 2003: Aiieee! 37068 exited with error code: 11 2003-06-11T11:41:48 ERROR(200) zdaemon zdaemon: Wed Jun 11 11:41:48 2003: Aiieee! 47611 exited with error code: 11 2003-06-11T11:55:27 ERROR(200) zdaemon zdaemon: Wed Jun 11 11:55:27 2003: Aiieee! 49437 exited with error code: 11 2003-06-11T11:58:59 ERROR(200) zdaemon zdaemon: Wed Jun 11 11:58:59 2003: Aiieee! 89916 exited with error code: 11 2003-06-11T12:11:12 ERROR(200) zdaemon zdaemon: Wed Jun 11 12:11:12 2003: Aiieee! 99043 exited with error code: 11 2003-06-11T12:12:13 ERROR(200) zdaemon zdaemon: Wed Jun 11 12:12:13 2003: Aiieee! 23124 exited with error code: 11 2003-06-11T12:13:03 ERROR(200) zdaemon zdaemon: Wed Jun 11 12:13:03 2003: Aiieee! 24129 exited with error code: 11 2003-06-11T12:15:32 ERROR(200) zdaemon zdaemon: Wed Jun 11 12:15:32 2003: Aiieee! 24993 exited with error code: 11 2003-06-11T12:19:16 ERROR(200) zdaemon zdaemon: Wed Jun 11 12:19:16 2003: Aiieee! 34656 exited with error code: 11 2003-06-11T14:40:21 ERROR(200) zdaemon zdaemon: Wed Jun 11 14:40:21 2003: Aiieee! 34298 exited with error code: 11 2003-06-11T15:57:04 ERROR(200) zdaemon zdaemon: Wed Jun 11 15:57:04 2003: Aiieee! 58796 exited with error code: 11 2003-06-11T16:38:18 ERROR(200) zdaemon zdaemon: Wed Jun 11 16:38:18 2003: Aiieee! 38035 exited with error code: 11 2003-06-11T19:10:26 ERROR(200) zdaemon zdaemon: Wed Jun 11 19:10:26 2003: Aiieee! 50725 exited with error code: 11 2003-06-11T19:45:28 ERROR(200) zdaemon zdaemon: Wed Jun 11 19:45:28 2003: Aiieee! 99307 exited with error code: 11 2003-06-11T20:03:25 ERROR(200) zdaemon zdaemon: Wed Jun 11 20:03:25 2003: Aiieee! 67697 exited with error code: 11 2003-06-11T20:03:55 ERROR(200) zdaemon zdaemon: Wed Jun 11 20:03:55 2003: Aiieee! 87865 exited with error code: 11 2003-06-11T20:05:20 ERROR(200) zdaemon zdaemon: Wed Jun 11 20:05:20 2003: Aiieee! 88080 exited with error code: 11 2003-06-11T20:13:52 ERROR(200) zdaemon zdaemon: Wed Jun 11 20:13:52 2003: Aiieee! 90481 exited with error code: 11

June11

6:55pm (EST) - set stack to <<24, re-make, etc.

more log::

2003-06-12T00:49:17 ERROR(200) zdaemon zdaemon: Thu Jun 12 00:49:17 2003: Aiieee! 40806 exited with error code: 11 2003-06-12T03:12:54 ERROR(200) zdaemon zdaemon: Thu Jun 12 03:12:54 2003: Aiieee! 64987 exited with error code: 10 2003-06-12T06:01:01 ERROR(200) zdaemon zdaemon: Thu Jun 12 06:01:01 2003: Aiieee! 3676 exited with error code: 11 2003-06-12T09:15:15 ERROR(200) zdaemon zdaemon: Thu Jun 12 09:15:15 2003: Aiieee! 14809 exited with error code: 4 2003-06-12T11:22:56 ERROR(200) zdaemon zdaemon: Thu Jun 12 11:22:56 2003: Aiieee! 34245 exited with error code: 11 2003-06-12T11:24:08 ERROR(200) zdaemon zdaemon: Thu Jun 12 11:24:08 2003: Aiieee! 56868 exited with error code: 11 2003-06-12T15:38:53 ERROR(200) zdaemon zdaemon: Thu Jun 12 15:38:53 2003: Aiieee! 59566 exited with error code: 11 2003-06-12T16:34:25 ERROR(200) zdaemon zdaemon: Thu Jun 12 16:34:25 2003: Aiieee! 49073 exited with error code: 11 2003-06-12T17:24:17 ERROR(200) zdaemon zdaemon: Thu Jun 12 17:24:17 2003: Aiieee! 97412 exited with error code: 11 2003-06-12T19:15:58 ERROR(200) zdaemon zdaemon: Thu Jun 12 19:15:58 2003: Aiieee! 27400 exited with error code: 11 2003-06-12T19:16:27 ERROR(200) zdaemon zdaemon: Thu Jun 12 19:16:27 2003: Aiieee! 36694 exited with error code: 11 2003-06-12T23:18:04 ERROR(200) zdaemon zdaemon: Thu Jun 12 23:18:04 2003: Aiieee! 36911 exited with error code: 11 2003-06-12T23:19:13 ERROR(200) zdaemon zdaemon: Thu Jun 12 23:19:13 2003: Aiieee! 34863 exited with error code: 11 2003-06-13T03:17:20 ERROR(200) zdaemon zdaemon: Fri Jun 13 03:17:20 2003: Aiieee! 37097 exited with error code: 11 2003-06-13T04:39:56 ERROR(200) zdaemon zdaemon: Fri Jun 13 04:39:56 2003: Aiieee! 40716 exited with error code: 11 2003-06-13T06:12:28 ERROR(200) zdaemon zdaemon: Fri Jun 13 06:12:28 2003: Aiieee! 79391 exited with error code: 11 2003-06-13T10:51:46 ERROR(200) zdaemon zdaemon: Fri Jun 13 10:51:46 2003: Aiieee! 47307 exited with error code: 4 2003-06-13T10:53:00 ERROR(200) zdaemon zdaemon: Fri Jun 13 10:53:00 2003: Aiieee! 3315 exited with error code: 11 2003-06-13T10:53:57 ERROR(200) zdaemon zdaemon: Fri Jun 13 10:53:57 2003: Aiieee! 6013 exited with error code: 11

memory snapshot --BillSeitz, 2003/06/13 13:08 GMT
per Zwiki:SiteReliability try::

ps -maxwO vsize |grep /usr/local/zope |grep -v grep |awk '{print $2}'|sort -nr

Not getting exact output, but do get ::

57380 7764 5144 2512

another memory snapshot --BillSeitz, 2003/06/13 19:45 GMT
top line 56756

a snapshot with more info --BillSeitz, 2003/06/13 19:58 GMT
ps -maxwO vsize |grep zope |grep -v grep |sort -nr +1 ::

83036 56756 ?? SNJ 3:43.90 /usr/local/bin/python /usr/local/zope/2.5.1/z2.py -W 9800 -w 8080 -f 21 40804 7764 ?? SJ 1:43.29 /usr/local/bin/python /usr/local/zope/2.5.1/lib/python/ZEO/start.py -U /usr/local/zope/instance/var 40805 5144 ?? Is J 0:00.13 /usr/local/bin/python /usr/local/zope/2.5.1/z2.py -W 9800 -w 8080 -f 21 40802 2512 ?? Is J 0:00.00 /usr/local/bin/python /usr/local/zope/2.5.1/lib/python/ZEO/start.py -U /usr/local/zope/instance/var

restricted number threads --BillSeitz, 2003/06/14 12:12 GMT
edited 'zope_start' to include '-t 2'

which Python is running --BillSeitz, 2003/06/14 13:48 GMT
Simon says: Your ps shows zope using /usr/local/bin/python

So I did::

imeme# cat zope_start #! /bin/sh INSTANCE_HOME=/usr/local/zope/instance export INSTANCE_HOME STUPID_LOG_FILE=$INSTANCE_HOME/var/events.log export STUPID_LOG_FILE exec /usr/local/bin/python
/usr/local/zope/2.5.1/z2.py
-W 9800
-w 8080
-f 21
-t 2
"$@"

imeme# cd /usr/local/bin imeme# ls -l python* -r-xr-xr-x 1 root wheel 518908 Apr 14 2002 python -rwxr-xr-x 1 root wheel 1091 Oct 17 2002 python.menu -rwxrwxrwx 1 root wheel 469444 Jun 12 2001 python1.5 -r-xr-xr-x 1 root wheel 521428 Jun 8 02:41 python2.1 -rwxr-xr-x 1 root wheel 983 Oct 17 2002 python_installer -rwxr-xr-x 1 root wheel 1031 Feb 12 2002 python_upgrade

imeme# cd /usr/ports/lang/python21/work/Python-2.1.3 imeme# ls -l python* -rwxr-xr-x 1 root wheel 611931 Jun 8 02:13 python

Duh, also looked at ZMI Control Panel: Python Version: 2.1.3 (#1, Apr 14 2002, 01:34:47) (GCC 2.95.3 20010315 (release) (FreeBSD))

My guess here is that the 'python' being run is from Apr2002, not the new one. Correct?

If that's right, should I (a) delete python and (b) copy python2.1 python?

a weirder concern --BillSeitz, 2003/06/15 11:02 GMT
if I changed the stack size to 24 on Jun11, then why do none of the Pythons have a Jun11 mod-date?

some improvements in stability --BillSeitz, 2003/06/15 11:06 GMT
This morning, have been up 12 hours. Because of the '-t 2'?

copied over Python --BillSeitz, 2003/06/16 12:52 GMT
Jon Ringuette confirmed that I needed to re-copy over python. I tried to do a 'make install', to make sure I had the latest stack size (24), but I accidentally did a 'make all' instead. Which seemed to crap out partway through. Then, having confirmed that the correct theads file was still there, I tried a 'make install', which didn't work either. So I just went to the /bin/ folder, 'rm python' and 'cp python2.1 python', which worked, then started up, which worked. We'll see.

a single page where editing causes a crash --BillSeitz, 2003/06/16 13:40 GMT
see Chaos Magick - but that's the "old" version - having added a couple paragraphs, I can no longer save the page - each attempt causes the server to crash. (see 'new' Chaos Magic.txt file on my laptop) (confirmed from events.log that it's still 'error code: 11')

will leave it alone for awhile --BillSeitz, 2003/06/16 13:59 GMT
Just crashed it again by trying to save Chaos Magick. Will leave it alone for a few hours and see whether it dies on its own.

hmmm.. --2003/06/16 20:21 GMT
Tried opening/saving your Chaos Magick page on my FreeBSD 5.0/Plone 1.0.1/Zwiki test server.

Zope Version (Zope 2.6.1 (source release, python 2.1, linux2), python 2.1.3, freebsd5)

Python Version 2.1.3 (#1, Feb 19 2003, 15:57:15) [GCC 3.2.1 [FreeBSD] 20021119 (release)]

System Platform freebsd5

Zwiki: 0.16.

My quick guess is that the page edit is forcing Zwiki/Zope/Python over the 100Mb imeme limit.

But why?

Pieter B

hmmm... cannot edit your page --2003/06/16 20:23 GMT
Hmm... I didn't find any errors while editing on my test server.

I found a problem saving a edited document on your server (I needed to authenticate myself).

Pieter B

now I really screwed things up --BillSeitz, 2003/06/17 17:57 GMT
Since my Jun16 activities didn't buy me anything, and Simon's binary didn't seem to actually be available, I thought I'd try something more drastic.

Since I couldn't seem to do a 'make all' the other day, I thought maybe I needed to dump stuff that was in that directory. So I went into /ports/ and did a 'rm -r *' and immediately said "oh shit, you know that directory was completely empty in the first place".

Decided to try again anyway. Saw I had the python*.tar file still, so untarred it, which worked. But then tried 'make all' and got immediate errors due to missing '/ports/Mk/'. Probably time for a phone call.

rebuilding ports directory --BillSeitz, 2003/06/20 17:05 GMT
follow <http://imeme.net/wiki/Cvs U P> - it worked

then untar the python bits again (Jun7)

make failed --BillSeitz, 2003/06/20 17:42 GMT
tried 'cd /usr/ports/lang/python21' - then both 'make all' and 'make install' failed with message of 'no COMMENT variable defined'

try 'make' in different directory --BillSeitz, 2003/06/20 17:44 GMT
moved to '/usr/ports/lang/python' instead and try 'make all'

where do things really stand? --BillSeitz, 2003/06/21 00:39 GMT
The 'make all' seemed to run

When I look in '/usr/local/bin/' none of the Pythons are from today.

When I look in '/usr/ports/lang/python21/' there's no 'work' subdirectory.

So it looks like things really aren't made yet.

I look at '/usr/ports/lang/python21/files/patch-Python-thread_pthread.h-stacksize' and see it still sets stacksize to 17, so I know I still have to over-write that file.

looks like there's a new version --BillSeitz, 2003/06/23 13:29 GMT
inside '/usr/ports/lang/python/work/' I find '/Python-2.2.3' - can I run this Zope with this Python? Would rather not try.

So what do I do about /python21/? Decide to try to get it to work.

So back into that directory, try just 'make'. Get the msg about lack of 'comment' variable. Research and add a line to the 'Makefile'.

Then get msg about existence of deprecated 'pkg-comment' file in the directory, so rename it. Now 'make' works.

Next: reconsider the process of patching the pthreads file for stacksize...

patching --BillSeitz, 2003/06/23 13:55 GMT
It seemed weird to me that I could change a .h file and then just 'make install' - plus the dates seemed wrong when I was looking inside the /bin/ directory, etc. But this confirmed I could change the .h file and then just 'make install', so I'll believe them.

'cd /usr/ports/lang/python21/work/Python-2.1.3/Python' - there's a thead_pthread.h file there with a new date, but it still says stacksize 17 inside, so I edit it.

'cd /usr/ports/lang/python21/' then ZMI to shut down zope, then 'killall -9 python' then 'make install' which seems to work. So do the startup commands, and everything seems to be running.

But is the right one running yet? --BillSeitz, 2003/06/23 14:02 GMT
ZMI still shows Python ' 2.1.3 (#1, Jun 8 2003, 02:13:38)'

'cd /usr/local/bin' and 'ls -l python*' shows no new python!

'cd /usr/ports/lang/python21/work/Python-2.1.3' finds a python built yesterday - so what about today's tweak to the thread_pthread.h file? Goddammit this is tiresome.

maybe that was it --BillSeitz, 2003/06/23 14:40 GMT
(Duh, lost earlier post because tried to Append when server was down - hitting back button does no good in that situation)

Based on Zwiki:Patch Python 213 tried doing 'make install FORCE_PKG_REGISTER=1' which didn't help, then tried it after a 'make deinstall' which at least did some stuff, but with the same files left in the end.

Then noticed some stuff from the spewage::

copying build/lib.freebsd-4.8-RELEASE-i386-2.1/zlib.so -> /home/pieterb/tmp/python213patched/lib/python2.1/lib-dynload running install_scripts not copying build/scripts/pydoc (output up-to-date) changing mode of /home/pieterb/tmp/python213patched/bin/pydoc to 100755 warning: install: modules installed to '/home/pieterb/tmp/python213patched/lib/python2.1/lib-dynload/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself ===> Generating temporary packing list install -c -o root -g wheel -m 444 /usr/ports/lang/python21/work/Python-2.1.3/Misc/python.man /home/pieterb/tmp/python213patched/man/man1/python2.1.1 ===> Compressing manual pages for python-2.1.3 ===> Registering installation for python-2.1.3 ===> SECURITY REPORT: This port has installed the following files which may act as network servers and may therefore pose a remote security risk to the system. /home/pieterb/tmp/python213patched/lib/python2.1/lib-dynload/_socket.so

  If there are vulnerabilities in these programs there may be a security
  risk to the system. FreeBSD makes no guarantee about the security of
  ports included in the Ports Collection. Please type 'make deinstall'
  to deinstall the port if this is a concern.

So there's some action in a '/home/pieterb/' directory. So dig around in there and find a python from today, so copy that over to '/usr/local/bin/'

But, then again, ZMI still shows Python ' 2.1.3 (#1, Jun 22 2003, 13:26:03)'

But at this point I'm tired so I'll wait and see what happens.

key open question --BillSeitz, 2003/06/23 14:42 GMT
If I'm working with a python built within '/home/pieterb/' then is it using the thread_pthread.h file I tweaked?

some progress --BillSeitz, 2003/06/24 11:58 GMT
Server was up for 20hrs!

But... saving that [change](http://webseitz.fluxent.com/articles/Chaos Magick.txt) to Chaos Magick crashed it again!

Up 7 hrs since that crash --BillSeitz, 2003/06/24 19:21 GMT
Am about to increase # of items on Front Page from 10 to 20.

That seemed OK --BillSeitz, 2003/06/24 22:12 GMT
But the server's only been up for 1hr40min, so it must have restarted at 4:30pm EDT

looking for Pieter B's code files --BillSeitz, 2003/06/24 22:14 GMT
Searched '/home/pieterb/' for thread*.h and didn't find anything.

try Chaos Magick page on zwiki.org --BillSeitz, 2003/06/24 22:25 GMT
add to bottom of Zwiki:BillSeitz and it saves with no problem. Is it my python, or my customized regexps?

OK, let's rebuild again --BillSeitz, 2003/06/25 12:16 GMT
Empty '/usr/ports/lang/python21/'

The link for the specific ports package on Zwiki:Patch Python 213 is bad, so research and pick ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/i386/packages-stable/All/python-2.1.3_3.tgz

Look inside and see no full path defined, so copy into the python21 directory. Then uncompress.

Don't see any 'make' files. There is a compiled file within 'bin'. But there is no 'thread_pthread.h' anywhere.

Zope's been up 17hrs by the way --BillSeitz, 2003/06/25 14:23 GMT
which would go back to 6pm yesterday. (But now something ugly's happening at the network level - tracert craps out somewhere in the middle of pnap.net ... ok now back up)

looking in python21 --BillSeitz, 2003/06/25 14:27 GMT
'find' doesn't catch any 'makefile' but it does find a 'makesetup' which appears to generate a makefile (didn't run it, just looked at the contents). But what options do I set?

Simon helped me hack a solution --BillSeitz, 2003/06/25 16:43 GMT
Gave me a copy of his bigstack file, which we put in place. Was able to save the Chaos Magick page now! So now we wait and watch...

Fresh problems --BillSeitz, 2004/03/20 09:45 EST

Server's been very unstable for the last week or so.

  • one thing that reliably crashes the server is grabbing Front Page with Python urllib2.urlopen - even though viewing the page via browser just before or after works fine.

    • hmmm, maybe some other Web Robot has a similar effect? I've blocked a lot of the pointless ones in the last month, but I just noticed a new one yesterday, maybe that's worth trying...

No changes in software. So probably a resource-quantity issue.

  • though should probably do some sniffing around to see whether anything has been automatically changed at a low level of the stack, or something.

For now just changed Front Page to show 20 items instead of 30.

Hmm, maybe new hosting client has built big site quickly, and I haven't put a catalog in? Nope, nobody's got much.

May10 - dump of 'Aiiee' grep from May9 - 53 in the day, some almost every hour::

2004-05-09T00:25:47 ERROR(200) zdaemon zdaemon: Sun May 9 00:25:47 2004: Aiieee! 18086 exited with error code: 6 2004-05-09T00:56:35 ERROR(200) zdaemon zdaemon: Sun May 9 00:56:35 2004: Aiieee! 71799 exited with error code: 11 2004-05-09T01:06:55 ERROR(200) zdaemon zdaemon: Sun May 9 01:06:55 2004: Aiieee! 96311 exited with error code: 11 2004-05-09T01:25:51 ERROR(200) zdaemon zdaemon: Sun May 9 01:25:51 2004: Aiieee! 5876 exited with error code: 10 2004-05-09T01:36:23 ERROR(200) zdaemon zdaemon: Sun May 9 01:36:23 2004: Aiieee! 21886 exited with error code: 11 2004-05-09T01:42:38 ERROR(200) zdaemon zdaemon: Sun May 9 01:42:38 2004: Aiieee! 30057 exited with error code: 10 2004-05-09T02:20:13 ERROR(200) zdaemon zdaemon: Sun May 9 02:20:13 2004: Aiieee! 34519 exited with error code: 11 2004-05-09T02:39:35 ERROR(200) zdaemon zdaemon: Sun May 9 02:39:35 2004: Aiieee! 63806 exited with error code: 6 2004-05-09T02:53:34 ERROR(200) zdaemon zdaemon: Sun May 9 02:53:34 2004: Aiieee! 79526 exited with error code: 11 2004-05-09T03:24:14 ERROR(200) zdaemon zdaemon: Sun May 9 03:24:14 2004: Aiieee! 91781 exited with error code: 6 2004-05-09T03:34:04 ERROR(200) zdaemon zdaemon: Sun May 9 03:34:04 2004: Aiieee! 30179 exited with error code: 11 2004-05-09T03:37:26 ERROR(200) zdaemon zdaemon: Sun May 9 03:37:26 2004: Aiieee! 38309 exited with error code: 11 2004-05-09T04:24:53 ERROR(200) zdaemon zdaemon: Sun May 9 04:24:53 2004: Aiieee! 41762 exited with error code: 6 2004-05-09T05:19:29 ERROR(200) zdaemon zdaemon: Sun May 9 05:19:29 2004: Aiieee! 79663 exited with error code: 6 2004-05-09T05:22:43 ERROR(200) zdaemon zdaemon: Sun May 9 05:22:43 2004: Aiieee! 25121 exited with error code: 11 2004-05-09T05:33:57 ERROR(200) zdaemon zdaemon: Sun May 9 05:33:57 2004: Aiieee! 29445 exited with error code: 6 2004-05-09T06:09:37 ERROR(200) zdaemon zdaemon: Sun May 9 06:09:37 2004: Aiieee! 37754 exited with error code: 6 2004-05-09T07:22:40 ERROR(200) zdaemon zdaemon: Sun May 9 07:22:40 2004: Aiieee! 67245 exited with error code: 11 2004-05-09T07:28:16 ERROR(200) zdaemon zdaemon: Sun May 9 07:28:16 2004: Aiieee! 29930 exited with error code: 6 2004-05-09T07:48:10 ERROR(200) zdaemon zdaemon: Sun May 9 07:48:10 2004: Aiieee! 33971 exited with error code: 6 2004-05-09T08:12:06 ERROR(200) zdaemon zdaemon: Sun May 9 08:12:06 2004: Aiieee! 51126 exited with error code: 10 2004-05-09T08:29:04 ERROR(200) zdaemon zdaemon: Sun May 9 08:29:04 2004: Aiieee! 72717 exited with error code: 6 2004-05-09T08:31:46 ERROR(200) zdaemon zdaemon: Sun May 9 08:31:46 2004: Aiieee! 86318 exited with error code: 11 2004-05-09T09:40:08 ERROR(200) zdaemon zdaemon: Sun May 9 09:40:08 2004: Aiieee! 90587 exited with error code: 6 2004-05-09T09:54:32 ERROR(200) zdaemon zdaemon: Sun May 9 09:54:32 2004: Aiieee! 52121 exited with error code: 6 2004-05-09T10:24:52 ERROR(200) zdaemon zdaemon: Sun May 9 10:24:52 2004: Aiieee! 62109 exited with error code: 6 2004-05-09T10:34:46 ERROR(200) zdaemon zdaemon: Sun May 9 10:34:46 2004: Aiieee! 88956 exited with error code: 11 2004-05-09T10:35:26 ERROR(200) zdaemon zdaemon: Sun May 9 10:35:26 2004: Aiieee! 96831 exited with error code: 11 2004-05-09T10:45:31 ERROR(200) zdaemon zdaemon: Sun May 9 10:45:31 2004: Aiieee! 99887 exited with error code: 6 2004-05-09T10:51:10 ERROR(200) zdaemon zdaemon: Sun May 9 10:51:10 2004: Aiieee! 8933 exited with error code: 11 2004-05-09T10:55:28 ERROR(200) zdaemon zdaemon: Sun May 9 10:55:28 2004: Aiieee! 13057 exited with error code: 11 2004-05-09T10:56:00 ERROR(200) zdaemon zdaemon: Sun May 9 10:56:00 2004: Aiieee! 16903 exited with error code: 11 2004-05-09T11:26:50 ERROR(200) zdaemon zdaemon: Sun May 9 11:26:50 2004: Aiieee! 16970 exited with error code: 6 2004-05-09T13:47:26 ERROR(200) zdaemon zdaemon: Sun May 9 13:47:26 2004: Aiieee! 41663 exited with error code: 11 2004-05-09T14:37:29 ERROR(200) zdaemon zdaemon: Sun May 9 14:37:29 2004: Aiieee! 59753 exited with error code: 11 2004-05-09T15:50:17 ERROR(200) zdaemon zdaemon: Sun May 9 15:50:17 2004: Aiieee! 3588 exited with error code: 11 2004-05-09T16:56:35 ERROR(200) zdaemon zdaemon: Sun May 9 16:56:35 2004: Aiieee! 64984 exited with error code: 11 2004-05-09T18:42:21 ERROR(200) zdaemon zdaemon: Sun May 9 18:42:21 2004: Aiieee! 19309 exited with error code: 6 2004-05-09T18:46:48 ERROR(200) zdaemon zdaemon: Sun May 9 18:46:48 2004: Aiieee! 8509 exited with error code: 10 2004-05-09T18:50:41 ERROR(200) zdaemon zdaemon: Sun May 9 18:50:41 2004: Aiieee! 12546 exited with error code: 11 2004-05-09T19:19:36 ERROR(200) zdaemon zdaemon: Sun May 9 19:19:36 2004: Aiieee! 16580 exited with error code: 10 2004-05-09T19:35:47 ERROR(200) zdaemon zdaemon: Sun May 9 19:35:47 2004: Aiieee! 38789 exited with error code: 11 2004-05-09T19:37:47 ERROR(200) zdaemon zdaemon: Sun May 9 19:37:47 2004: Aiieee! 55170 exited with error code: 11 2004-05-09T19:42:52 ERROR(200) zdaemon zdaemon: Sun May 9 19:42:52 2004: Aiieee! 55496 exited with error code: 11 2004-05-09T20:00:15 ERROR(200) zdaemon zdaemon: Sun May 9 20:00:15 2004: Aiieee! 59574 exited with error code: 10 2004-05-09T20:24:51 ERROR(200) zdaemon zdaemon: Sun May 9 20:24:51 2004: Aiieee! 73191 exited with error code: 6 2004-05-09T20:36:55 ERROR(200) zdaemon zdaemon: Sun May 9 20:36:55 2004: Aiieee! 94572 exited with error code: 11 2004-05-09T20:47:59 ERROR(200) zdaemon zdaemon: Sun May 9 20:47:59 2004: Aiieee! 6692 exited with error code: 6 2004-05-09T20:48:26 ERROR(200) zdaemon zdaemon: Sun May 9 20:48:26 2004: Aiieee! 15309 exited with error code: 11 2004-05-09T21:04:22 ERROR(200) zdaemon zdaemon: Sun May 9 21:04:22 2004: Aiieee! 15433 exited with error code: 11 2004-05-09T22:11:31 ERROR(200) zdaemon zdaemon: Sun May 9 22:11:31 2004: Aiieee! 28286 exited with error code: 11 2004-05-09T23:24:25 ERROR(200) zdaemon zdaemon: Sun May 9 23:24:25 2004: Aiieee! 84855 exited with error code: 6 2004-05-09T23:26:23 ERROR(200) zdaemon zdaemon: Sun May 9 23:26:23 2004: Aiieee! 43829 exited with error code: 6


Edited:    |       |    Search Twitter for discussion