Zwiki Freebsd Stability Problems (WebSeitz/wikilog)

WebSeitz/wikilog

Zwiki Freebsd Stability Problems

Whenever any Form of Government becomes destructive of these ends, it is the Right of the People to alter or to abolish it, and to institute new Government, laying its foundation on such principles and organizing its powers in such form, as to them shall seem most likely to effect their Safety and Happiness.

(backlinks off)

(map off)

(search off)

last edited by BillSeitz on Jul 2, 2008 7:28 am

It turns out there are problems with the configuration of PyThon for FreeBSD, which tends to result in instability, in particular in ZWiki installations (which use lots of Regular Expression parsing). Here's my log. (For reference will track some Unix Commands.)

Context: going from...

going from zope 2.3.2 (python 1.5.2) w/ zwiki 0.9.5
to zope 2.5.1 (python 2.1.3) w/ zwiki 0.17

June 5, 2003

Got things "up", but consistently getting Proxy Error 502 when trying to view Front Page. A "normal" page is still immediately viewable, so I don't think zope is actually crashed.

Lots of pages that don't make much sense to me yet.
Quick hack: I noticed my target cache size was 400, even though I had 3200+ objects across caches. (And 15,727 objects in the database.) So I upped the target to 3300. Immediately the number in the cache turned to 5334. One Simon page notes that you can get 2+ threads, and the cache will be multiplied by the threads. That didn't make the page viewable.
Tried changing Front Page to just show 20 items instead of 50, but can't even save a change.
Just went to [Control Panel] to try restarting service, notice that it says it's only been up for 1 min! So maybe it is crashing all the time, but it restarts fast.
- yes - I try for Front Page, it waits 10 seconds, give me the Proxy Error, then I immediately look at [Control Panel] and see that the server's only been up for 5 seconds.
to reduce size of Front Page, take local version hack to only show 20, then export/copy/import. That seems to be working for now.
- a bit later, it died again
also run a Cache [Full Sweep], just for the heck of it.
- looked 15 minutes later, see 3250 items in log, so it's stuck with a single thread
look at /usr/local/zope/instance/var/events.log (tail -1000 events.log)
- see 2003-06-05T16:35:06 [ERROR](200) zdaemon zdaemon: Thu Jun 5 16:35:06 2003: Aiieee! 36154 exited with error code: 11
- which leads to http://zwiki.org/IssueNo0162BrowsingDiffsSomeOtherThingsCauseZopeToCrashWithAieeeErrorCode11SIGSEGV and http://zwiki.org/IssueNo0226OnFreeBSDPythonMayCrashRepeatablyWhenViewingOrSavingCertainWikiPages
my system eventually was so locked I couldn't get into anything. I called iMeme and they whacked it (the emergency zope restart button on the support pages should do the trick next time).
- I tried talking to them at IRC, but nobody was listening, though some were logged on: irc.imeme.net /join #imeme
- tried to kill using: killall -9 python
  - Then start your Zope by running
  - /usr/local/zope/instance/storage_start
  - /usr/local/zope/instance/zope_start
- I removed the My Blog Roll from Front Page to reduce the regexp load. Then shortened My Recent Key Pages also.
  - this probably isn't sufficient, since I just checked the [Control Panel] and saw that the server just restarted again.
- Jon @ iMeme says he's already made the FreeBSD python patch! So I posted at zwiki.org and emailed Simon.
- status
  - at 6:21pm, it's been up 1h11m
  - at 10pm, it's only been up 18m, so it rebooted at least once
- hmm, I wonder whether changing the page type would help? Of course, can't do it through the [ZMI] because of the Zope Get Slice Bug!
  - hmm, something broke the rendering. Will have to look at this later, including the pageTypes of more static wiki pages.

Jun7

looks like Jon's patch doesn't include the larger stack size
http://zwiki.org/PatchPython213
Jon says You can cvsup the ports tree and just have it build the version of python you need from that its the same setup we use though I'm not sure the date on the python version I installed. The only stack increase I did was the one offered on the Plone site.
not sure what I'm doing, but I'll try to get at least partway
look in /usr/ports/ - there's no lang/
- try to create with [WinSCP], no permissions
- try to do via ssh - have to change to su
download file to laptop, copy to imeme - have to do chown
do tar -x
- says can't open /dev/sa0
try unzipping on my machine, won't let me because of a funny file name (at least when I do it auto)
- should probably rename the file, copy to imeme, then rename again
  - argh, can't rename file within [WinZip]!
notice that files have full root set, so maybe need to do untar from root directory.
after gnashing, realize need to use gunzip on a .tgz file - that works, generating a .tar file
but "tar x usr-....*.tar" still doesn't work - gives "not in archive" message.
- need to use f modifier to specify tar file name, so 'tar xvf [[filename]]' - that works!
- some confusion: some of document says /usr/ports/langs... others say /usr/ports/lang...' - the file uses the singular, so it just re-created some semi-parallel directories
do make extract, it says "bad value or [PYTHON]_[VERSION]: python2.1. Use one of python1.5, python1.6 or python2.0 (default)... Error code 1
Jon says I needed a new CVS base directory, which he gave me. Now I can type : make all - then : kill off all running pythons (killall -9 python) - then : make install
- make all
- download .h file , change <<17 to <<20, upload (to work/Python-2.1.3/Python/) (chown again)
- then killall, then make-install
- then storage_startup, then zope_startup
- up and running at 10:45pm
- still up at 7am!!!

Jun8

stability looking good

whoops, nope, looks like it rebooted at 1:30pm.

Want to download log file and check changes in frequency of reboot. (note times are 2 hours ahead of [EST]) :

 2003-06-05T05:44:11 ERROR(200) zdaemon zdaemon: Thu Jun  5 05:44:11 2003: Aiieee! 56733 exited with error code: 11
2003-06-05T05:44:32 ERROR(200) zdaemon zdaemon: Thu Jun  5 05:44:32 2003: Aiieee! 68608 exited with error code: 11
2003-06-05T05:45:25 ERROR(200) zdaemon zdaemon: Thu Jun  5 05:45:25 2003: Aiieee! 68656 exited with error code: 11
2003-06-05T05:46:37 ERROR(200) zdaemon zdaemon: Thu Jun  5 05:46:37 2003: Aiieee! 69036 exited with error code: 11
2003-06-05T05:59:25 ERROR(200) zdaemon zdaemon: Thu Jun  5 05:59:25 2003: Aiieee! 69210 exited with error code: 11
2003-06-05T06:15:11 ERROR(200) zdaemon zdaemon: Thu Jun  5 06:15:11 2003: Aiieee! 72207 exited with error code: 11
2003-06-05T06:15:26 ERROR(200) zdaemon zdaemon: Thu Jun  5 06:15:26 2003: Aiieee! 76591 exited with error code: 11
2003-06-05T07:00:31 ERROR(200) zdaemon zdaemon: Thu Jun  5 07:00:31 2003: Aiieee! 76622 exited with error code: 11
2003-06-05T07:15:26 ERROR(200) zdaemon zdaemon: Thu Jun  5 07:15:26 2003: Aiieee! 87731 exited with error code: 11
2003-06-05T07:22:13 ERROR(200) zdaemon zdaemon: Thu Jun  5 07:22:13 2003: Aiieee! 91673 exited with error code: 11
2003-06-05T07:29:25 ERROR(200) zdaemon zdaemon: Thu Jun  5 07:29:25 2003: Aiieee! 93250 exited with error code: 11
2003-06-05T07:29:46 ERROR(200) zdaemon zdaemon: Thu Jun  5 07:29:46 2003: Aiieee! 94866 exited with error code: 11
2003-06-05T07:43:58 ERROR(200) zdaemon zdaemon: Thu Jun  5 07:43:58 2003: Aiieee! 94909 exited with error code: 11
2003-06-05T07:44:27 ERROR(200) zdaemon zdaemon: Thu Jun  5 07:44:27 2003: Aiieee! 98524 exited with error code: 11
2003-06-05T07:44:56 ERROR(200) zdaemon zdaemon: Thu Jun  5 07:44:56 2003: Aiieee! 98609 exited with error code: 11
2003-06-05T07:45:43 ERROR(200) zdaemon zdaemon: Thu Jun  5 07:45:43 2003: Aiieee! 98662 exited with error code: 11
2003-06-05T07:46:33 ERROR(200) zdaemon zdaemon: Thu Jun  5 07:46:33 2003: Aiieee! 99034 exited with error code: 11
2003-06-05T07:58:45 ERROR(200) zdaemon zdaemon: Thu Jun  5 07:58:45 2003: Aiieee! 99160 exited with error code: 11
2003-06-05T08:12:47 ERROR(200) zdaemon zdaemon: Thu Jun  5 08:12:47 2003: Aiieee! 2080 exited with error code: 11
2003-06-05T08:15:18 ERROR(200) zdaemon zdaemon: Thu Jun  5 08:15:18 2003: Aiieee! 6464 exited with error code: 11
2003-06-05T09:15:25 ERROR(200) zdaemon zdaemon: Thu Jun  5 09:15:25 2003: Aiieee! 6850 exited with error code: 11
2003-06-05T09:28:24 ERROR(200) zdaemon zdaemon: Thu Jun  5 09:28:24 2003: Aiieee! 27340 exited with error code: 11
2003-06-05T09:28:49 ERROR(200) zdaemon zdaemon: Thu Jun  5 09:28:49 2003: Aiieee! 30404 exited with error code: 11
2003-06-05T09:32:15 ERROR(200) zdaemon zdaemon: Thu Jun  5 09:32:15 2003: Aiieee! 30455 exited with error code: 11
2003-06-05T09:32:33 ERROR(200) zdaemon zdaemon: Thu Jun  5 09:32:33 2003: Aiieee! 31515 exited with error code: 11
2003-06-05T09:37:13 ERROR(200) zdaemon zdaemon: Thu Jun  5 09:37:13 2003: Aiieee! 32560 exited with error code: 11
2003-06-05T14:02:33 ERROR(200) zdaemon zdaemon: Thu Jun  5 14:02:33 2003: Aiieee! 32634 exited with error code: 11
2003-06-05T14:15:28 ERROR(200) zdaemon zdaemon: Thu Jun  5 14:15:28 2003: Aiieee! 98995 exited with error code: 11
2003-06-05T15:08:59 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:08:59 2003: Aiieee! 2418 exited with error code: 11
2003-06-05T15:09:16 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:09:16 2003: Aiieee! 17020 exited with error code: 11
2003-06-05T15:09:31 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:09:31 2003: Aiieee! 17103 exited with error code: 11
2003-06-05T15:15:25 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:15:25 2003: Aiieee! 17134 exited with error code: 11
2003-06-05T15:23:08 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:23:08 2003: Aiieee! 18767 exited with error code: 11
2003-06-05T15:28:06 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:28:06 2003: Aiieee! 20467 exited with error code: 11
2003-06-05T15:31:11 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:31:11 2003: Aiieee! 21576 exited with error code: 11
2003-06-05T15:35:46 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:35:46 2003: Aiieee! 22553 exited with error code: 11
2003-06-05T15:36:13 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:36:13 2003: Aiieee! 23557 exited with error code: 11
2003-06-05T15:44:13 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:44:13 2003: Aiieee! 23658 exited with error code: 11
2003-06-05T15:49:04 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:49:04 2003: Aiieee! 25583 exited with error code: 11
2003-06-05T15:59:50 ERROR(200) zdaemon zdaemon: Thu Jun  5 15:59:50 2003: Aiieee! 28591 exited with error code: 11
2003-06-05T16:03:15 ERROR(200) zdaemon zdaemon: Thu Jun  5 16:03:15 2003: Aiieee! 29260 exited with error code: 11
2003-06-05T16:07:56 ERROR(200) zdaemon zdaemon: Thu Jun  5 16:07:56 2003: Aiieee! 30366 exited with error code: 11
2003-06-05T16:11:13 ERROR(200) zdaemon zdaemon: Thu Jun  5 16:11:13 2003: Aiieee! 31384 exited with error code: 11
2003-06-05T16:25:10 ERROR(200) zdaemon zdaemon: Thu Jun  5 16:25:10 2003: Aiieee! 32383 exited with error code: 11
2003-06-05T16:35:06 ERROR(200) zdaemon zdaemon: Thu Jun  5 16:35:06 2003: Aiieee! 36154 exited with error code: 11
2003-06-05T16:36:41 ERROR(200) zdaemon zdaemon: Thu Jun  5 16:36:41 2003: Aiieee! 38170 exited with error code: 11
2003-06-05T16:37:29 ERROR(200) zdaemon zdaemon: Thu Jun  5 16:37:29 2003: Aiieee! 38404 exited with error code: 11
2003-06-05T16:42:35 ERROR(200) zdaemon zdaemon: Thu Jun  5 16:42:35 2003: Aiieee! 38527 exited with error code: 11
2003-06-05T16:51:24 ERROR(200) zdaemon zdaemon: Thu Jun  5 16:51:24 2003: Aiieee! 39833 exited with error code: 11
2003-06-05T19:04:53 ERROR(200) zdaemon zdaemon: Thu Jun  5 19:04:53 2003: Aiieee! 42057 exited with error code: 11
2003-06-05T19:06:18 ERROR(200) zdaemon zdaemon: Thu Jun  5 19:06:18 2003: Aiieee! 75618 exited with error code: 11
2003-06-05T19:06:51 ERROR(200) zdaemon zdaemon: Thu Jun  5 19:06:51 2003: Aiieee! 76006 exited with error code: 11
2003-06-05T19:20:55 ERROR(200) zdaemon zdaemon: Thu Jun  5 19:20:55 2003: Aiieee! 76068 exited with error code: 11
2003-06-05T19:22:02 ERROR(200) zdaemon zdaemon: Thu Jun  5 19:22:02 2003: Aiieee! 79687 exited with error code: 11
2003-06-05T19:22:19 ERROR(200) zdaemon zdaemon: Thu Jun  5 19:22:19 2003: Aiieee! 79922 exited with error code: 11
2003-06-05T19:22:44 ERROR(200) zdaemon zdaemon: Thu Jun  5 19:22:44 2003: Aiieee! 79954 exited with error code: 11
2003-06-05T20:21:38 ERROR(200) zdaemon zdaemon: Thu Jun  5 20:21:38 2003: Aiieee! 94768 exited with error code: 11
2003-06-05T20:22:57 ERROR(200) zdaemon zdaemon: Thu Jun  5 20:22:57 2003: Aiieee! 94942 exited with error code: 11
2003-06-05T20:23:32 ERROR(200) zdaemon zdaemon: Thu Jun  5 20:23:32 2003: Aiieee! 95121 exited with error code: 11
2003-06-05T20:57:45 ERROR(200) zdaemon zdaemon: Thu Jun  5 20:57:45 2003: Aiieee! 95217 exited with error code: 11
2003-06-05T20:59:56 ERROR(200) zdaemon zdaemon: Thu Jun  5 20:59:56 2003: Aiieee! 4085 exited with error code: 11
2003-06-05T21:09:35 ERROR(200) zdaemon zdaemon: Thu Jun  5 21:09:35 2003: Aiieee! 5172 exited with error code: 11
2003-06-06T01:58:48 ERROR(200) zdaemon zdaemon: Fri Jun  6 01:58:48 2003: Aiieee! 34782 exited with error code: 11
2003-06-06T01:59:23 ERROR(200) zdaemon zdaemon: Fri Jun  6 01:59:23 2003: Aiieee! 86989 exited with error code: 11
2003-06-06T03:20:09 ERROR(200) zdaemon zdaemon: Fri Jun  6 03:20:09 2003: Aiieee! 87740 exited with error code: 11
2003-06-06T03:36:14 ERROR(200) zdaemon zdaemon: Fri Jun  6 03:36:14 2003: Aiieee! 28285 exited with error code: 11
2003-06-06T10:36:38 ERROR(200) zdaemon zdaemon: Fri Jun  6 10:36:38 2003: Aiieee! 84711 exited with error code: 11
2003-06-06T13:26:37 ERROR(200) zdaemon zdaemon: Fri Jun  6 13:26:37 2003: Aiieee! 97747 exited with error code: 11
2003-06-06T15:55:37 ERROR(200) zdaemon zdaemon: Fri Jun  6 15:55:37 2003: Aiieee! 99765 exited with error code: 11
2003-06-06T16:43:04 ERROR(200) zdaemon zdaemon: Fri Jun  6 16:43:04 2003: Aiieee! 77825 exited with error code: 11
2003-06-06T17:01:59 ERROR(200) zdaemon zdaemon: Fri Jun  6 17:01:59 2003: Aiieee! 19829 exited with error code: 11
2003-06-06T17:03:03 ERROR(200) zdaemon zdaemon: Fri Jun  6 17:03:03 2003: Aiieee! 66566 exited with error code: 11
2003-06-06T20:38:53 ERROR(200) zdaemon zdaemon: Fri Jun  6 20:38:53 2003: Aiieee! 68304 exited with error code: 11
2003-06-06T21:45:13 ERROR(200) zdaemon zdaemon: Fri Jun  6 21:45:13 2003: Aiieee! 8498 exited with error code: 11
2003-06-06T22:07:15 ERROR(200) zdaemon zdaemon: Fri Jun  6 22:07:15 2003: Aiieee! 2081 exited with error code: 11
2003-06-06T22:08:08 ERROR(200) zdaemon zdaemon: Fri Jun  6 22:08:08 2003: Aiieee! 65099 exited with error code: 11
2003-06-06T22:11:59 ERROR(200) zdaemon zdaemon: Fri Jun  6 22:11:59 2003: Aiieee! 69105 exited with error code: 11
2003-06-07T02:27:50 ERROR(200) zdaemon zdaemon: Sat Jun  7 02:27:50 2003: Aiieee! 77106 exited with error code: 11
2003-06-07T02:29:29 ERROR(200) zdaemon zdaemon: Sat Jun  7 02:29:29 2003: Aiieee! 40743 exited with error code: 11
2003-06-07T05:52:57 ERROR(200) zdaemon zdaemon: Sat Jun  7 05:52:57 2003: Aiieee! 44512 exited with error code: 11
2003-06-07T15:30:17 ERROR(200) zdaemon zdaemon: Sat Jun  7 15:30:17 2003: Aiieee! 19608 exited with error code: 11
2003-06-07T17:38:38 ERROR(200) zdaemon zdaemon: Sat Jun  7 17:38:38 2003: Aiieee! 35806 exited with error code: 11
2003-06-07T17:39:06 ERROR(200) zdaemon zdaemon: Sat Jun  7 17:39:06 2003: Aiieee! 8202 exited with error code: 11
2003-06-07T18:08:28 ERROR(200) zdaemon zdaemon: Sat Jun  7 18:08:28 2003: Aiieee! 8497 exited with error code: 11
2003-06-07T23:20:11 ERROR(200) zdaemon zdaemon: Sat Jun  7 23:20:11 2003: Aiieee! 88073 exited with error code: 11
2003-06-07T23:24:19 ERROR(200) zdaemon zdaemon: Sat Jun  7 23:24:19 2003: Aiieee! 74714 exited with error code: 11
2003-06-08T00:47:33 ERROR(200) zdaemon zdaemon: Sun Jun  8 00:47:33 2003: Aiieee! 86720 exited with error code: 11
2003-06-09T00:45:41 ERROR(200) zdaemon zdaemon: Mon Jun  9 00:45:41 2003: Aiieee! 18785 exited with error code: 11
2003-06-09T11:28:05 ERROR(200) zdaemon zdaemon: Mon Jun  9 11:28:05 2003: Aiieee! 63639 exited with error code: 11
2003-06-09T11:31:36 ERROR(200) zdaemon zdaemon: Mon Jun  9 11:31:36 2003: Aiieee! 59055 exited with error code: 11
2003-06-09T11:32:52 ERROR(200) zdaemon zdaemon: Mon Jun  9 11:32:52 2003: Aiieee! 68023 exited with error code: 11
2003-06-09T12:34:42 ERROR(200) zdaemon zdaemon: Mon Jun  9 12:34:42 2003: Aiieee! 72762 exited with error code: 11
2003-06-09T16:40:23 ERROR(200) zdaemon zdaemon: Mon Jun  9 16:40:23 2003: Aiieee! 13875 exited with error code: 11
2003-06-10T07:39:44 ERROR(200) zdaemon zdaemon: Tue Jun 10 07:39:44 2003: Aiieee! 11385 exited with error code: 11
2003-06-10T12:42:58 ERROR(200) zdaemon zdaemon: Tue Jun 10 12:42:58 2003: Aiieee! 21168 exited with error code: 11
2003-06-10T16:50:56 ERROR(200) zdaemon zdaemon: Tue Jun 10 16:50:56 2003: Aiieee! 34937 exited with error code: 11
2003-06-10T18:07:47 ERROR(200) zdaemon zdaemon: Tue Jun 10 18:07:47 2003: Aiieee! 28985 exited with error code: 11
2003-06-10T18:15:26 ERROR(200) zdaemon zdaemon: Tue Jun 10 18:15:26 2003: Aiieee! 37789 exited with error code: 11
2003-06-10T18:46:20 ERROR(200) zdaemon zdaemon: Tue Jun 10 18:46:20 2003: Aiieee! 48744 exited with error code: 11
2003-06-10T18:48:11 ERROR(200) zdaemon zdaemon: Tue Jun 10 18:48:11 2003: Aiieee! 92211 exited with error code: 11
2003-06-10T18:51:44 ERROR(200) zdaemon zdaemon: Tue Jun 10 18:51:44 2003: Aiieee! 94985 exited with error code: 11
2003-06-10T22:33:09 ERROR(200) zdaemon zdaemon: Tue Jun 10 22:33:09 2003: Aiieee! 2346 exited with error code: 11
2003-06-10T23:09:13 ERROR(200) zdaemon zdaemon: Tue Jun 10 23:09:13 2003: Aiieee! 66535 exited with error code: 11
2003-06-10T23:54:58 ERROR(200) zdaemon zdaemon: Tue Jun 10 23:54:58 2003: Aiieee! 30495 exited with error code: 11
2003-06-10T23:55:28 ERROR(200) zdaemon zdaemon: Tue Jun 10 23:55:28 2003: Aiieee! 6541 exited with error code: 11
2003-06-10T23:59:50 ERROR(200) zdaemon zdaemon: Tue Jun 10 23:59:50 2003: Aiieee! 6955 exited with error code: 11
2003-06-11T00:07:55 ERROR(200) zdaemon zdaemon: Wed Jun 11 00:07:55 2003: Aiieee! 20424 exited with error code: 11
2003-06-11T00:09:29 ERROR(200) zdaemon zdaemon: Wed Jun 11 00:09:29 2003: Aiieee! 43123 exited with error code: 11
2003-06-11T00:15:14 ERROR(200) zdaemon zdaemon: Wed Jun 11 00:15:14 2003: Aiieee! 46657 exited with error code: 11
2003-06-11T01:20:54 ERROR(200) zdaemon zdaemon: Wed Jun 11 01:20:54 2003: Aiieee! 47907 exited with error code: 11
2003-06-11T01:21:29 ERROR(200) zdaemon zdaemon: Wed Jun 11 01:21:29 2003: Aiieee! 53993 exited with error code: 11
2003-06-11T01:28:52 ERROR(200) zdaemon zdaemon: Wed Jun 11 01:28:52 2003: Aiieee! 54397 exited with error code: 11
2003-06-11T01:34:43 ERROR(200) zdaemon zdaemon: Wed Jun 11 01:34:43 2003: Aiieee! 77124 exited with error code: 11
2003-06-11T01:46:26 ERROR(200) zdaemon zdaemon: Wed Jun 11 01:46:26 2003: Aiieee! 94865 exited with error code: 11
2003-06-11T02:38:07 ERROR(200) zdaemon zdaemon: Wed Jun 11 02:38:07 2003: Aiieee! 30051 exited with error code: 11
2003-06-11T02:51:42 ERROR(200) zdaemon zdaemon: Wed Jun 11 02:51:42 2003: Aiieee! 76694 exited with error code: 11
2003-06-11T02:52:41 ERROR(200) zdaemon zdaemon: Wed Jun 11 02:52:41 2003: Aiieee! 11463 exited with error code: 11
2003-06-11T03:09:58 ERROR(200) zdaemon zdaemon: Wed Jun 11 03:09:58 2003: Aiieee! 12908 exited with error code: 11
2003-06-11T09:17:52 ERROR(200) zdaemon zdaemon: Wed Jun 11 09:17:52 2003: Aiieee! 84378 exited with error code: 11
2003-06-11T09:44:50 ERROR(200) zdaemon zdaemon: Wed Jun 11 09:44:50 2003: Aiieee! 2721 exited with error code: 11
2003-06-11T10:15:16 ERROR(200) zdaemon zdaemon: Wed Jun 11 10:15:16 2003: Aiieee! 48833 exited with error code: 11
2003-06-11T11:40:37 ERROR(200) zdaemon zdaemon: Wed Jun 11 11:40:37 2003: Aiieee! 37068 exited with error code: 11
2003-06-11T11:41:48 ERROR(200) zdaemon zdaemon: Wed Jun 11 11:41:48 2003: Aiieee! 47611 exited with error code: 11
2003-06-11T11:55:27 ERROR(200) zdaemon zdaemon: Wed Jun 11 11:55:27 2003: Aiieee! 49437 exited with error code: 11
2003-06-11T11:58:59 ERROR(200) zdaemon zdaemon: Wed Jun 11 11:58:59 2003: Aiieee! 89916 exited with error code: 11
2003-06-11T12:11:12 ERROR(200) zdaemon zdaemon: Wed Jun 11 12:11:12 2003: Aiieee! 99043 exited with error code: 11
2003-06-11T12:12:13 ERROR(200) zdaemon zdaemon: Wed Jun 11 12:12:13 2003: Aiieee! 23124 exited with error code: 11
2003-06-11T12:13:03 ERROR(200) zdaemon zdaemon: Wed Jun 11 12:13:03 2003: Aiieee! 24129 exited with error code: 11
2003-06-11T12:15:32 ERROR(200) zdaemon zdaemon: Wed Jun 11 12:15:32 2003: Aiieee! 24993 exited with error code: 11
2003-06-11T12:19:16 ERROR(200) zdaemon zdaemon: Wed Jun 11 12:19:16 2003: Aiieee! 34656 exited with error code: 11
2003-06-11T14:40:21 ERROR(200) zdaemon zdaemon: Wed Jun 11 14:40:21 2003: Aiieee! 34298 exited with error code: 11
2003-06-11T15:57:04 ERROR(200) zdaemon zdaemon: Wed Jun 11 15:57:04 2003: Aiieee! 58796 exited with error code: 11
2003-06-11T16:38:18 ERROR(200) zdaemon zdaemon: Wed Jun 11 16:38:18 2003: Aiieee! 38035 exited with error code: 11
2003-06-11T19:10:26 ERROR(200) zdaemon zdaemon: Wed Jun 11 19:10:26 2003: Aiieee! 50725 exited with error code: 11
2003-06-11T19:45:28 ERROR(200) zdaemon zdaemon: Wed Jun 11 19:45:28 2003: Aiieee! 99307 exited with error code: 11
2003-06-11T20:03:25 ERROR(200) zdaemon zdaemon: Wed Jun 11 20:03:25 2003: Aiieee! 67697 exited with error code: 11
2003-06-11T20:03:55 ERROR(200) zdaemon zdaemon: Wed Jun 11 20:03:55 2003: Aiieee! 87865 exited with error code: 11
2003-06-11T20:05:20 ERROR(200) zdaemon zdaemon: Wed Jun 11 20:05:20 2003: Aiieee! 88080 exited with error code: 11
2003-06-11T20:13:52 ERROR(200) zdaemon zdaemon: Wed Jun 11 20:13:52 2003: Aiieee! 90481 exited with error code: 11

June11

6:55pm ([EST]) - set stack to <<24, re-make, etc.

more log:

 2003-06-12T00:49:17 ERROR(200) zdaemon zdaemon: Thu Jun 12 00:49:17 2003: Aiieee! 40806 exited with error code: 11
2003-06-12T03:12:54 ERROR(200) zdaemon zdaemon: Thu Jun 12 03:12:54 2003: Aiieee! 64987 exited with error code: 10
2003-06-12T06:01:01 ERROR(200) zdaemon zdaemon: Thu Jun 12 06:01:01 2003: Aiieee! 3676 exited with error code: 11
2003-06-12T09:15:15 ERROR(200) zdaemon zdaemon: Thu Jun 12 09:15:15 2003: Aiieee! 14809 exited with error code: 4
2003-06-12T11:22:56 ERROR(200) zdaemon zdaemon: Thu Jun 12 11:22:56 2003: Aiieee! 34245 exited with error code: 11
2003-06-12T11:24:08 ERROR(200) zdaemon zdaemon: Thu Jun 12 11:24:08 2003: Aiieee! 56868 exited with error code: 11
2003-06-12T15:38:53 ERROR(200) zdaemon zdaemon: Thu Jun 12 15:38:53 2003: Aiieee! 59566 exited with error code: 11
2003-06-12T16:34:25 ERROR(200) zdaemon zdaemon: Thu Jun 12 16:34:25 2003: Aiieee! 49073 exited with error code: 11
2003-06-12T17:24:17 ERROR(200) zdaemon zdaemon: Thu Jun 12 17:24:17 2003: Aiieee! 97412 exited with error code: 11
2003-06-12T19:15:58 ERROR(200) zdaemon zdaemon: Thu Jun 12 19:15:58 2003: Aiieee! 27400 exited with error code: 11
2003-06-12T19:16:27 ERROR(200) zdaemon zdaemon: Thu Jun 12 19:16:27 2003: Aiieee! 36694 exited with error code: 11
2003-06-12T23:18:04 ERROR(200) zdaemon zdaemon: Thu Jun 12 23:18:04 2003: Aiieee! 36911 exited with error code: 11
2003-06-12T23:19:13 ERROR(200) zdaemon zdaemon: Thu Jun 12 23:19:13 2003: Aiieee! 34863 exited with error code: 11
2003-06-13T03:17:20 ERROR(200) zdaemon zdaemon: Fri Jun 13 03:17:20 2003: Aiieee! 37097 exited with error code: 11
2003-06-13T04:39:56 ERROR(200) zdaemon zdaemon: Fri Jun 13 04:39:56 2003: Aiieee! 40716 exited with error code: 11
2003-06-13T06:12:28 ERROR(200) zdaemon zdaemon: Fri Jun 13 06:12:28 2003: Aiieee! 79391 exited with error code: 11
2003-06-13T10:51:46 ERROR(200) zdaemon zdaemon: Fri Jun 13 10:51:46 2003: Aiieee! 47307 exited with error code: 4
2003-06-13T10:53:00 ERROR(200) zdaemon zdaemon: Fri Jun 13 10:53:00 2003: Aiieee! 3315 exited with error code: 11
2003-06-13T10:53:57 ERROR(200) zdaemon zdaemon: Fri Jun 13 10:53:57 2003: Aiieee! 6013 exited with error code: 11

memory snapshot --Bill Seitz, 2003/06/13 13:08 [GMT]
per ZWiki:SiteReliability try:

  ps -maxwO vsize |grep /usr/local/zope |grep -v grep |awk '{print $2}'|sort -nr

Not getting exact output, but do get :

another memory snapshot --Bill Seitz, 2003/06/13 19:45 [GMT]
top line 56756

a snapshot with more info --Bill Seitz, 2003/06/13 19:58 [GMT]
ps -maxwO vsize |grep zope |grep -v grep |sort -nr +1 :

 83036 56756  ??  SNJ    3:43.90 /usr/local/bin/python /usr/local/zope/2.5.1/z2.py -W 9800 -w 8080 -f 21
40804  7764  ??  SJ     1:43.29 /usr/local/bin/python /usr/local/zope/2.5.1/lib/python/ZEO/start.py -U /usr/local/zope/instance/var
40805  5144  ??  IsJ    0:00.13 /usr/local/bin/python /usr/local/zope/2.5.1/z2.py -W 9800 -w 8080 -f 21
40802  2512  ??  IsJ    0:00.00 /usr/local/bin/python /usr/local/zope/2.5.1/lib/python/ZEO/start.py -U /usr/local/zope/instance/var

restricted number threads --Bill Seitz, 2003/06/14 12:12 [GMT]
edited zope_start to include -t 2

which Python is running --Bill Seitz, 2003/06/14 13:48 [GMT]
Simon says: Your ps shows zope using /usr/local/bin/python

So I did:

 imeme# cat zope_start
#! /bin/sh
INSTANCE_HOME=/usr/local/zope/instance
export INSTANCE_HOME
STUPID_LOG_FILE=$INSTANCE_HOME/var/events.log
export STUPID_LOG_FILE
exec /usr/local/bin/python \
     /usr/local/zope/2.5.1/z2.py \
     -W 9800 \
     -w 8080 \
     -f 21 \
     -t 2 \
     "$@"
-----------
imeme# cd /usr/local/bin
imeme# ls -l python*
-r-xr-xr-x  1 root  wheel  518908 Apr 14  2002 python
-rwxr-xr-x  1 root  wheel    1091 Oct 17  2002 python.menu
-rwxrwxrwx  1 root  wheel  469444 Jun 12  2001 python1.5
-r-xr-xr-x  1 root  wheel  521428 Jun  8 02:41 python2.1
-rwxr-xr-x  1 root  wheel     983 Oct 17  2002 python_installer
-rwxr-xr-x  1 root  wheel    1031 Feb 12  2002 python_upgrade
--------------
imeme# cd /usr/ports/lang/python21/work/Python-2.1.3
imeme# ls -l python*
-rwxr-xr-x  1 root  wheel  611931 Jun  8 02:13 python

Duh, also looked at [ZMI] [Control Panel]: Python Version: 2.1.3 (#1, Apr 14 2002, 01:34:47) ([GCC] 2.95.3 20010315 (release) (FreeBSD))

My guess here is that the python being run is from Apr2002, not the new one. Correct?

If that's right, should I (a) delete python and (b) copy python2.1 python?

a weirder concern --Bill Seitz, 2003/06/15 11:02 [GMT]
if I changed the stack size to 24 on Jun11, then why do none of the Pythons have a Jun11 mod-date?

some improvements in stability --Bill Seitz, 2003/06/15 11:06 [GMT]
This morning, have been up 12 hours. Because of the -t 2?

copied over Python --Bill Seitz, 2003/06/16 12:52 [GMT]
[Jon Ringuette] confirmed that I needed to re-copy over python. I tried to do a make install, to make sure I had the latest stack size (24), but I accidentally did a make all instead. Which seemed to crap out partway through. Then, having confirmed that the correct theads file was still there, I tried a make install, which didn't work either. So I just went to the /bin/ folder, rm python and cp python2.1 python, which worked, then started up, which worked. We'll see.

a single page where editing causes a crash --Bill Seitz, 2003/06/16 13:40 [GMT]
see Chaos Magick - but that's the "old" version - having added a couple paragraphs, I can no longer save the page - each attempt causes the server to crash. (see new [Chaos Magic].txt file on my laptop) (confirmed from events.log that it's still error code: 11)

will leave it alone for awhile --Bill Seitz, 2003/06/16 13:59 [GMT]
Just crashed it again by trying to save Chaos Magick. Will leave it alone for a few hours and see whether it dies on its own.

hmmm.. --2003/06/16 20:21 [GMT]
Tried opening/saving your Chaos Magick page on my FreeBSD 5.0/Plone 1.0.1/Zwiki test server.

Zope Version (Zope 2.6.1 (source release, python 2.1, linux2), python 2.1.3, freebsd5)

Python Version 2.1.3 (#1, Feb 19 2003, 15:57:15) [[ G C C 3.2.1 [ Free B S D]] 20021119 (release)]

System Platform freebsd5

Zwiki: 0.16.

My quick guess is that the page edit is forcing Zwiki/Zope/Python over the 100Mb imeme limit.

But why?

[PieterB]

hmmm... cannot edit your page --2003/06/16 20:23 [GMT]
Hmm... I didn't find any errors while editing on my test server.

I found a problem saving a edited document on your server (I needed to authenticate myself).

[PieterB]

now I really screwed things up --Bill Seitz, 2003/06/17 17:57 [GMT]
Since my Jun16 activities didn't buy me anything, and Simon's binary didn't seem to actually be available, I thought I'd try something more drastic.

Since I couldn't seem to do a make all the other day, I thought maybe I needed to dump stuff that was in that directory. So I went into /ports/ and did a rm -r * and immediately said "oh shit, you know that directory was completely empty in the first place".

Decided to try again anyway. Saw I had the python*.tar file still, so untarred it, which worked. But then tried make all and got immediate errors due to missing /ports/Mk/. Probably time for a phone call.

rebuilding ports directory --Bill Seitz, 2003/06/20 17:05 [GMT]
follow http://imeme.net/wiki/CvsUP - it worked

then untar the python bits again (Jun7)

make failed --Bill Seitz, 2003/06/20 17:42 [GMT]
tried cd /usr/ports/lang/python21 - then both make all and make install failed with message of no COMMENT variable defined

try make in different directory --Bill Seitz, 2003/06/20 17:44 [GMT]
moved to /usr/ports/lang/python instead and try make all

where do things really stand? --Bill Seitz, 2003/06/21 00:39 [GMT]
The make all seemed to run

When I look in /usr/local/bin/ none of the Pythons are from today.

When I look in /usr/ports/lang/python21/ there's no work subdirectory.

So it looks like things really aren't made yet.

I look at /usr/ports/lang/python21/files/patch-Python-thread_pthread.h-stacksize and see it still sets stacksize to 17, so I know I still have to over-write that file.

looks like there's a new version --Bill Seitz, 2003/06/23 13:29 [GMT]
inside /usr/ports/lang/python/work/ I find /Python-2.2.3 - can I run this Zope with this Python? Would rather not try.

So what do I do about /python21/? Decide to try to get it to work.

So back into that directory, try just make. Get the msg about lack of comment variable. Research and add a line to the Makefile.

Then get msg about existence of deprecated pkg-comment file in the directory, so rename it. Now make works.

Next: reconsider the process of patching the pthreads file for stacksize...

patching --Bill Seitz, 2003/06/23 13:55 [GMT]
It seemed weird to me that I could change a .h file and then just make install - plus the dates seemed wrong when I was looking inside the /bin/ directory, etc. But this confirmed I could change the .h file and then just make install, so I'll believe them.

cd /usr/ports/lang/python21/work/Python-2.1.3/Python - there's a thead_pthread.h file there with a new date, but it still says stacksize 17 inside, so I edit it.

cd /usr/ports/lang/python21/ then [ZMI] to shut down zope, then killall -9 python then make install which seems to work. So do the startup commands, and everything seems to be running.

But is the right one running yet? --Bill Seitz, 2003/06/23 14:02 [GMT]
[ZMI] still shows Python 2.1.3 (#1, Jun 8 2003, 02:13:38)

cd /usr/local/bin and ls -l python* shows no new python!

cd /usr/ports/lang/python21/work/Python-2.1.3 finds a python built yesterday - so what about today's tweak to the thread_pthread.h file? Goddammit this is tiresome.

maybe that was it --Bill Seitz, 2003/06/23 14:40 [GMT]
(Duh, lost earlier post because tried to Append when server was down - hitting back button does no good in that situation)

Based on ZWiki:PatchPython213 tried doing make install FORCE_PKG_REGISTER=1 which didn't help, then tried it after a make deinstall which at least did some stuff, but with the same files left in the end.

Then noticed some stuff from the spewage:

 copying build/lib.freebsd-4.8-RELEASE-i386-2.1/zlib.so -> /home/pieterb/tmp/python213patched/lib/python2.1/lib-dynload
running install_scripts
not copying build/scripts/pydoc (output up-to-date)
changing mode of /home/pieterb/tmp/python213patched/bin/pydoc to 100755
warning: install: modules installed to '/home/pieterb/tmp/python213patched/lib/python2.1/lib-dynload/', which is not in Python's module search path (sys.path) -- you'll have to change the search path yourself
===>   Generating temporary packing list
install -c -o root -g wheel -m 444 /usr/ports/lang/python21/work/Python-2.1.3/Misc/python.man  /home/pieterb/tmp/python213patched/man/man1/python2.1.1
===>   Compressing manual pages for python-2.1.3
===>   Registering installation for python-2.1.3
===>  SECURITY REPORT:
      This port has installed the following files which may act as network
      servers and may therefore pose a remote security risk to the system.
/home/pieterb/tmp/python213patched/lib/python2.1/lib-dynload/_socket.so

      If there are vulnerabilities in these programs there may be a security
      risk to the system. FreeBSD makes no guarantee about the security of
      ports included in the Ports Collection. Please type 'make deinstall'
      to deinstall the port if this is a concern.

So there's some action in a /home/pieterb/ directory. So dig around in there and find a python from today, so copy that over to /usr/local/bin/

But, then again, [ZMI] still shows Python 2.1.3 (#1, Jun 22 2003, 13:26:03)

But at this point I'm tired so I'll wait and see what happens.

key open question --Bill Seitz, 2003/06/23 14:42 [GMT]
If I'm working with a python built within /home/pieterb/ then is it using the thread_pthread.h file I tweaked?

some progress --Bill Seitz, 2003/06/24 11:58 [GMT]
Server was up for 20hrs!

But... saving that change to Chaos Magick crashed it again!

Up 7 hrs since that crash --Bill Seitz, 2003/06/24 19:21 [GMT]
Am about to increase # of items on Front Page from 10 to 20.

That seemed [OK] --Bill Seitz, 2003/06/24 22:12 [GMT]
But the server's only been up for 1hr40min, so it must have restarted at 4:30pm [EDT]

looking for [PieterB]'s code files --Bill Seitz, 2003/06/24 22:14 [GMT]
Searched /home/pieterb/ for thread*.h and didn't find anything.

try Chaos Magick page on zwiki.org --Bill Seitz, 2003/06/24 22:25 [GMT]
add to bottom of ZWiki:BillSeitz and it saves with no problem. Is it my python, or my customized regexps?

[OK], let's rebuild again --Bill Seitz, 2003/06/25 12:16 [GMT]
Empty /usr/ports/lang/python21/

The link for the specific ports package on ZWiki:PatchPython213 is bad, so research and pick ftp://ftp.FreeBSD.org/pub/FreeBSD/ports/i386/packages-stable/All/python-2.1.3_3.tgz

Look inside and see no full path defined, so copy into the python21 directory. Then uncompress.

Don't see any make files. There is a compiled file within bin. But there is no thread_pthread.h anywhere.

Zope's been up 17hrs by the way --Bill Seitz, 2003/06/25 14:23 [GMT]
which would go back to 6pm yesterday. (But now something ugly's happening at the network level - tracert craps out somewhere in the middle of pnap.net ... ok now back up)

looking in python21 --Bill Seitz, 2003/06/25 14:27 [GMT]
find doesn't catch any makefile but it does find a makesetup which appears to generate a makefile (didn't run it, just looked at the contents). But what options do I set?

Simon helped me hack a solution --Bill Seitz, 2003/06/25 16:43 [GMT]
Gave me a copy of his bigstack file, which we put in place. Was able to save the Chaos Magick page now! So now we wait and watch...

Fresh problems --Bill Seitz, 2004/03/20 09:45 [EST]

Server's been very unstable for the last week or so.

one thing that reliably crashes the server is grabbing Front Page with PyThon urllib2.urlopen - even though viewing the page via browser just before or after works fine.
- hmmm, maybe some other Web Robot has a similar effect? I've blocked a lot of the pointless ones in the last month, but I just noticed a new one yesterday, maybe that's worth trying...

No changes in software. So probably a resource-quantity issue.

though should probably do some sniffing around to see whether anything has been automatically changed at a low level of the stack, or something.

For now just changed Front Page to show 20 items instead of 30.

Hmm, maybe new hosting client has built big site quickly, and I haven't put a catalog in? Nope, nobody's got much.

May10 - dump of Aiiee grep from May9 - 53 in the day, some almost every hour:

 2004-05-09T00:25:47 ERROR(200) zdaemon zdaemon: Sun May  9 00:25:47 2004: Aiieee! 18086 exited with error code: 6
2004-05-09T00:56:35 ERROR(200) zdaemon zdaemon: Sun May  9 00:56:35 2004: Aiieee! 71799 exited with error code: 11
2004-05-09T01:06:55 ERROR(200) zdaemon zdaemon: Sun May  9 01:06:55 2004: Aiieee! 96311 exited with error code: 11
2004-05-09T01:25:51 ERROR(200) zdaemon zdaemon: Sun May  9 01:25:51 2004: Aiieee! 5876 exited with error code: 10
2004-05-09T01:36:23 ERROR(200) zdaemon zdaemon: Sun May  9 01:36:23 2004: Aiieee! 21886 exited with error code: 11
2004-05-09T01:42:38 ERROR(200) zdaemon zdaemon: Sun May  9 01:42:38 2004: Aiieee! 30057 exited with error code: 10
2004-05-09T02:20:13 ERROR(200) zdaemon zdaemon: Sun May  9 02:20:13 2004: Aiieee! 34519 exited with error code: 11
2004-05-09T02:39:35 ERROR(200) zdaemon zdaemon: Sun May  9 02:39:35 2004: Aiieee! 63806 exited with error code: 6
2004-05-09T02:53:34 ERROR(200) zdaemon zdaemon: Sun May  9 02:53:34 2004: Aiieee! 79526 exited with error code: 11
2004-05-09T03:24:14 ERROR(200) zdaemon zdaemon: Sun May  9 03:24:14 2004: Aiieee! 91781 exited with error code: 6
2004-05-09T03:34:04 ERROR(200) zdaemon zdaemon: Sun May  9 03:34:04 2004: Aiieee! 30179 exited with error code: 11
2004-05-09T03:37:26 ERROR(200) zdaemon zdaemon: Sun May  9 03:37:26 2004: Aiieee! 38309 exited with error code: 11
2004-05-09T04:24:53 ERROR(200) zdaemon zdaemon: Sun May  9 04:24:53 2004: Aiieee! 41762 exited with error code: 6
2004-05-09T05:19:29 ERROR(200) zdaemon zdaemon: Sun May  9 05:19:29 2004: Aiieee! 79663 exited with error code: 6
2004-05-09T05:22:43 ERROR(200) zdaemon zdaemon: Sun May  9 05:22:43 2004: Aiieee! 25121 exited with error code: 11
2004-05-09T05:33:57 ERROR(200) zdaemon zdaemon: Sun May  9 05:33:57 2004: Aiieee! 29445 exited with error code: 6
2004-05-09T06:09:37 ERROR(200) zdaemon zdaemon: Sun May  9 06:09:37 2004: Aiieee! 37754 exited with error code: 6
2004-05-09T07:22:40 ERROR(200) zdaemon zdaemon: Sun May  9 07:22:40 2004: Aiieee! 67245 exited with error code: 11
2004-05-09T07:28:16 ERROR(200) zdaemon zdaemon: Sun May  9 07:28:16 2004: Aiieee! 29930 exited with error code: 6
2004-05-09T07:48:10 ERROR(200) zdaemon zdaemon: Sun May  9 07:48:10 2004: Aiieee! 33971 exited with error code: 6
2004-05-09T08:12:06 ERROR(200) zdaemon zdaemon: Sun May  9 08:12:06 2004: Aiieee! 51126 exited with error code: 10
2004-05-09T08:29:04 ERROR(200) zdaemon zdaemon: Sun May  9 08:29:04 2004: Aiieee! 72717 exited with error code: 6
2004-05-09T08:31:46 ERROR(200) zdaemon zdaemon: Sun May  9 08:31:46 2004: Aiieee! 86318 exited with error code: 11
2004-05-09T09:40:08 ERROR(200) zdaemon zdaemon: Sun May  9 09:40:08 2004: Aiieee! 90587 exited with error code: 6
2004-05-09T09:54:32 ERROR(200) zdaemon zdaemon: Sun May  9 09:54:32 2004: Aiieee! 52121 exited with error code: 6
2004-05-09T10:24:52 ERROR(200) zdaemon zdaemon: Sun May  9 10:24:52 2004: Aiieee! 62109 exited with error code: 6
2004-05-09T10:34:46 ERROR(200) zdaemon zdaemon: Sun May  9 10:34:46 2004: Aiieee! 88956 exited with error code: 11
2004-05-09T10:35:26 ERROR(200) zdaemon zdaemon: Sun May  9 10:35:26 2004: Aiieee! 96831 exited with error code: 11
2004-05-09T10:45:31 ERROR(200) zdaemon zdaemon: Sun May  9 10:45:31 2004: Aiieee! 99887 exited with error code: 6
2004-05-09T10:51:10 ERROR(200) zdaemon zdaemon: Sun May  9 10:51:10 2004: Aiieee! 8933 exited with error code: 11
2004-05-09T10:55:28 ERROR(200) zdaemon zdaemon: Sun May  9 10:55:28 2004: Aiieee! 13057 exited with error code: 11
2004-05-09T10:56:00 ERROR(200) zdaemon zdaemon: Sun May  9 10:56:00 2004: Aiieee! 16903 exited with error code: 11
2004-05-09T11:26:50 ERROR(200) zdaemon zdaemon: Sun May  9 11:26:50 2004: Aiieee! 16970 exited with error code: 6
2004-05-09T13:47:26 ERROR(200) zdaemon zdaemon: Sun May  9 13:47:26 2004: Aiieee! 41663 exited with error code: 11
2004-05-09T14:37:29 ERROR(200) zdaemon zdaemon: Sun May  9 14:37:29 2004: Aiieee! 59753 exited with error code: 11
2004-05-09T15:50:17 ERROR(200) zdaemon zdaemon: Sun May  9 15:50:17 2004: Aiieee! 3588 exited with error code: 11
2004-05-09T16:56:35 ERROR(200) zdaemon zdaemon: Sun May  9 16:56:35 2004: Aiieee! 64984 exited with error code: 11
2004-05-09T18:42:21 ERROR(200) zdaemon zdaemon: Sun May  9 18:42:21 2004: Aiieee! 19309 exited with error code: 6
2004-05-09T18:46:48 ERROR(200) zdaemon zdaemon: Sun May  9 18:46:48 2004: Aiieee! 8509 exited with error code: 10
2004-05-09T18:50:41 ERROR(200) zdaemon zdaemon: Sun May  9 18:50:41 2004: Aiieee! 12546 exited with error code: 11
2004-05-09T19:19:36 ERROR(200) zdaemon zdaemon: Sun May  9 19:19:36 2004: Aiieee! 16580 exited with error code: 10
2004-05-09T19:35:47 ERROR(200) zdaemon zdaemon: Sun May  9 19:35:47 2004: Aiieee! 38789 exited with error code: 11
2004-05-09T19:37:47 ERROR(200) zdaemon zdaemon: Sun May  9 19:37:47 2004: Aiieee! 55170 exited with error code: 11
2004-05-09T19:42:52 ERROR(200) zdaemon zdaemon: Sun May  9 19:42:52 2004: Aiieee! 55496 exited with error code: 11
2004-05-09T20:00:15 ERROR(200) zdaemon zdaemon: Sun May  9 20:00:15 2004: Aiieee! 59574 exited with error code: 10
2004-05-09T20:24:51 ERROR(200) zdaemon zdaemon: Sun May  9 20:24:51 2004: Aiieee! 73191 exited with error code: 6
2004-05-09T20:36:55 ERROR(200) zdaemon zdaemon: Sun May  9 20:36:55 2004: Aiieee! 94572 exited with error code: 11
2004-05-09T20:47:59 ERROR(200) zdaemon zdaemon: Sun May  9 20:47:59 2004: Aiieee! 6692 exited with error code: 6
2004-05-09T20:48:26 ERROR(200) zdaemon zdaemon: Sun May  9 20:48:26 2004: Aiieee! 15309 exited with error code: 11
2004-05-09T21:04:22 ERROR(200) zdaemon zdaemon: Sun May  9 21:04:22 2004: Aiieee! 15433 exited with error code: 11
2004-05-09T22:11:31 ERROR(200) zdaemon zdaemon: Sun May  9 22:11:31 2004: Aiieee! 28286 exited with error code: 11
2004-05-09T23:24:25 ERROR(200) zdaemon zdaemon: Sun May  9 23:24:25 2004: Aiieee! 84855 exited with error code: 6
2004-05-09T23:26:23 ERROR(200) zdaemon zdaemon: Sun May  9 23:26:23 2004: Aiieee! 43829 exited with error code: 6

See Back Links: z2003-06-11- Zwiki Freebsd Fixing |

User Options Recent Changes Help Page

Bill Seitz, fluxent at gmail dot com, Weblog