Apache Crashes Under Load With XCache

[starams5]

First, I want to say you guys did a good job with XCache 2.0.0 Final (XCache-2.0.0-php-5.3.10-Win32-VC9-x86). This version is more stable than the 1xx versions. I only have one problem, when the server starts to get under load (80+ users), Apache crashes and the Apache service will stops. This doesn't happen with APC or file based.

I've tried different settings, but I'm no export with these settings. Could you please help, and explain in detail what I can change to get this working.

Specs:
Windows Server 2008 R2
Duel Quad Core Xeons
16Gb Memory

XAMPP 1.7.7, which includes:

Apache 2.2.21
MySQL 5.5.16
PHP 5.3.8 (Memory Limit Set At 128M)

Thanks

[moo]

is your cache almost full before the cash? it is reported in #246 and is fixed. i'm gonna role out a release soon

[starams5]

thanks. i doubt the cache's were full. i tried 60M-512M, the results were always the same, Apache crash. it was hard going back to ACP after using XCache. the difference is noticeable. with XCache the site was faster & smoother until site trafic became heavy. i'll just wait for the next release, hopefully the problem is resolved.

thanks.

[starams5]

not sure if this will help or not. i uploaded an image of the Apache error log, see link:

 http://img545.imageshack.us/img545/7043/regioncapture.png

[moo]

please attach image here using "attach file". for error_log you can post text here directly

[starams5]

i've been testing on my test site using stress software, and the crashes continued (80+ users). then i tried the settings below, and so far no more crashes. before i move XCache to my live site, could you give your opinion on the settings below (all other settings default):

; to enable : xcache.size=64M etc (any size > 0) and your system mmap allows
xcache.size = 0M

[moo]

xcache.size = 0M means no cache. i'll try to stress it too but i wonder what php code can cause this issue. cause i sure don't have the same code as you do

[starams5]

Update. We may have solved the problem. I updated to PHP 5.4 (& XCache PHP 5.4 version). Updated the necessary .dll's, and took care of the errors in the Apache log. Tested on the test site with 140 users, 15 clicks per user, 20 second delay per user. No Apache crashes.

Moved everything to the live site, all of the above, no errors. If you could help me with the settings, I would appreciate it. I have a large site with a lot of files loading, and I think it should be a little faster than it is, the settings are probably wrong. Could you please recommend optimum settings. Thanks.

Current configuration:

[xcache-common]
;; install as zend extension (recommended), normally "$extension_dir/xcache.so"
;zend_extension = /usr/local/lib/php/extensions/non-debug-non-zts-xxx/xcache.so
; zend_extension_ts = /usr/local/lib/php/extensions/non-debug-zts-xxx/xcache.so
;; For windows users, replace xcache.so with php_xcache.dll
zend_extension_ts = c:/xampp/php/ext/php_xcache.dll
;; or install as extension, make sure your extension_dir setting is correct
; extension = xcache.so
;; or win32:
extension = php_xcache.dll

[xcache.admin]
xcache.admin.enable_auth = Off
xcache.admin.user = "runner"
; xcache.admin.pass = md5($your_password)
xcache.admin.pass = "5f4dcc3b5aa765d61d8327deb882cf99"

[xcache]
; ini only settings, all the values here is default unless explained

; select low level shm/allocator scheme implemenation
xcache.shm_scheme = "mmap"
; to disable: xcache.size=0
; to enable : xcache.size=64M etc (any size > 0) and your system mmap allows
xcache.size = 128M
; set to cpu count (cat /proc/cpuinfo |grep -c processor)
xcache.count = 1
; just a hash hints, you can always store count(items) > slots
xcache.slots = 8k
; ttl of the cache item, 0=forever
xcache.ttl = 300
; interval of gc scanning expired items, 0=no scan, other values is in seconds
xcache.gc_interval = 480

; same as aboves but for variable cache
xcache.var_size = 4M
xcache.var_count = 1
xcache.var_slots = 8K
; default ttl
xcache.var_ttl = 0
xcache.var_maxttl = 0
xcache.var_gc_interval = 300

; N/A for /dev/zero
xcache.readonly_protection = Off
; for *nix, xcache.mmap_path is a file path, not directory.
; Use something like "/tmp/xcache" if you want to turn on ReadonlyProtection
; 2 group of php won't share the same /tmp/xcache
; for win32, xcache.mmap_path=anonymous map name, not file path
xcache.mmap_path = "/dev/zero"

; leave it blank(disabled) or "/tmp/phpcore/"
; make sure it's writable by php (open_basedir is not checked)
xcache.coredump_directory = ""

; per request settings
xcache.cacher = On
xcache.stat = On
xcache.optimizer = Off

xcache.test = Off
xcache.experimental = Off

[xcache.coverager]
; per request settings
; enable coverage data collecting for xcache.coveragedump_directory and xcache_coverager_start/stop/get/clean() functions (will hurt executing performance)
xcache.coverager = Off

; ini only settings
; make sure it's readable (open_basedir is checked) by coverage viewer script
; requires xcache.coverager=On
xcache.coveragedump_directory = ""

[moo]

it seems "zend_extension_ts = c:/xampp/php/ext/php_xcache.dll" should be "zend_extension = c:/xampp/php/ext/php_xcache.dll" since PHP 5.4. why not put a phpinfo() page and see if xcache is loaded. and xcache.size = 128M may not enough if you have lots of php files. just put xcache/admin/* to your htdocs/xcache-admin/ to see how your cache is used

[starams5]

Apache Error Log

[moo]

what url do you request to get this error triggered so it is logged. is it the 1st time request after you start apache or the 2nd time? if it's easy to reproduce could you please make an short reproduce-able code?

[starams5]

thanks for your help. I have a media site, videos, audio, etc; I publish the updates to Facebook, Twittter, & a few more networks. This is when Apache crashes, when the people start coming to the site.

I'm working on the xcache admin now. .htaccess is not allowing access to xcache admin, trying to find a work-around.

Thanks.

[starams5]

XCache Admin Image

[starams5]

Okay, I got into the admin, please see attached file.

[moo]

for now you may have to disable XCache. within a few days, i'm gonna add mini dump support for PHP with XCache enabled to generate php.dmp before i can analyze the crash

[moo]

maybe fixed in [903]. i'll roll out an win32 build soon for your testing before i close this ticket.

[starams5]

Okay thanks. I'm still using xcache, but watching the situation closely. Although the errors appear in the log, its staying online for now. Its hard to go back to APC after using xcache. My site is slow with APC. I'll keep checking the site for the new build. Thanks.

[moo]

please download from latest snapshot from http://xcache.lighttpd.net/pub/snapshots/ for your favorite version. reopen this ticket if it's not fixed

[moo]

btw, if things go good. report. if things go bad, attach new error.log again. thanks for your helping to make XCache better

[starams5]

i sure will. either way, you're the best, man. thanks for all your help. i'll test when things slow down a bit. thanks.

[starams5]

I'm not sure what's going on, but my caching option will not except XCache-trunk-r903-php-5.4.3-nts-Win32-VC9-x86 (option to enable is grayed out). See attached screen-shots. Thanks.

Test site configuration:

[xcache-common]
;; install as zend extension (recommended), normally "$extension_dir/xcache.so"
;zend_extension = /usr/local/lib/php/extensions/non-debug-non-zts-xxx/xcache.so
; zend_extension_ts = /usr/local/lib/php/extensions/non-debug-zts-xxx/xcache.so
;; For windows users, replace xcache.so with php_xcache.dll
zend_extension_ts = f:/xampp/php/ext/php_xcache.dll
;; or install as extension, make sure your extension_dir setting is correct
; extension = xcache.so
;; or win32:
extension = php_xcache.dll

[xcache.admin]
xcache.admin.enable_auth = Off
xcache.admin.user = "runner"
; xcache.admin.pass = md5($your_password)
xcache.admin.pass = "5f4dcc3b5aa765d61d8327deb882cf99"

[xcache]
; ini only settings, all the values here is default unless explained

; select low level shm/allocator scheme implemenation
xcache.shm_scheme = "mmap"
; to disable: xcache.size=0
; to enable : xcache.size=64M etc (any size > 0) and your system mmap allows
xcache.size = 60M
; set to cpu count (cat /proc/cpuinfo |grep -c processor)
xcache.count = 1
; just a hash hints, you can always store count(items) > slots
xcache.slots = 8K
; ttl of the cache item, 0=forever
xcache.ttl = 0
; interval of gc scanning expired items, 0=no scan, other values is in seconds
xcache.gc_interval = 0

; same as aboves but for variable cache
xcache.var_size = 4M
xcache.var_count = 1
xcache.var_slots = 8K
; default ttl
xcache.var_ttl = 0
xcache.var_maxttl = 0
xcache.var_gc_interval = 300

; N/A for /dev/zero
xcache.readonly_protection = Off
; for *nix, xcache.mmap_path is a file path, not directory.
; Use something like "/tmp/xcache" if you want to turn on ReadonlyProtection
; 2 group of php won't share the same /tmp/xcache
; for win32, xcache.mmap_path=anonymous map name, not file path
xcache.mmap_path = "/dev/zero"

; leave it blank(disabled) or "/tmp/phpcore/"
; make sure it's writable by php (open_basedir is not checked)
xcache.coredump_directory = ""

; per request settings
xcache.cacher = On
xcache.stat = On
xcache.optimizer = Off

xcache.test = Off
xcache.experimental = Off

[xcache.coverager]
; per request settings
; enable coverage data collecting for xcache.coveragedump_directory and xcache_coverager_start/stop/get/clean() functions (will hurt executing performance)
xcache.coverager = Off

; ini only settings
; make sure it's readable (open_basedir is checked) by coverage viewer script
; requires xcache.coverager=On
xcache.coveragedump_directory = ""

[moo]

it may not load correctly. check phpinfo and/or xcache-admin page again

[starams5]

i've gone over the configuration several times, it works just fine with XCache-2.0.0-php-5.4.0-Win32-VC9-x86.

i tried swapping out the .dll with XCache-trunk-r903-php-5.4.3-nts-Win32-VC9-x86 after enabling XCache-2.0.0-php-5.4.0-Win32-VC9-x86 (re-started Apache), no luck. cant enter xcache admin, and the test site crashes. xcache admin work just fine with XCache-2.0.0-php-5.4.0-Win32-VC9-x86

see attachments. thanks.

[moo]

hrm... looks like i failed to build the ts version (the name without nts)

[moo]

do you have any IM so we can interact with each other in order to reduce time

[starams5]

thanks. i don't have any IM accounts. i just re-enabled the site chat bar, we can use that unless you have another solution. only problem is, you have to log into the site to use the bar. you can log in with Facebook or Twitter accounts. i'll add you as a friend, and we can chat. thanks.

 http://www.urbanepeeps.com/

[starams5]

i took care of everything, we can chat on the bar now. account activated. thanks.

[moo]

cannot login. no error prompted

[moo]

thanks for your time. i'll be available at irc.freenode.net #lighttpd 5 hours later from now. you may use  http://webchat.freenode.net/

i had to build non-zts first

[starams5]

okay thanks. also, i reset the password on your account at the site, and tested the login. it's been mailed to you. thanks.

[moo]

moo @ freenode.net#lighttpd, zts build uploaded. please try

[starams5]

thanks moo. do you have a link?

[moo]

same link at http://xcache.lighttpd.net/pub/snapshots/, the ones without -nts-

[starams5]

Hi Moo,

Apologizes for not getting back to you sooner, I still work part-time job. The new .dll worked! I will test for a while before moving it to the live site. Like I said, you're the best! No one else would have took the time to help.

Thanks

[moo]

when you done testing, make sure to report, no matter if it's buggy or not bug free

[starams5]

sure will. i moved it to the live site. its the only way to really test it, with live traffic. so far, so good. i'll report back tonight or in the morning. thanks again for all your help.

[starams5]

just checking back. no xcache/apache crashes. traffic got heavy yesterday, again, no crashes. i can't thank you enough. APC was okay until traffic got heavy, then the site got slow. with xcache, you can't even tell the traffic is heavy, the site remains fast & smooth with or without a load. bye bye APC!

[moo]

ok. thanks for your confirmation to the fix for windows zend thread safe (zts) enabled. it will be released soon as 2.0.1, which is more likely an identical version you're using now

[starams5]

how it going moo,

just an update. xcache has been performing well. i had one problem but i think that's been resolved. the bots started crashing apache, google to be specific. i raised the ThreadsPerChild? (per apache error log) and that seemed to have resolved the issue. xcache is much faster with the higher settings (ThreadsPerChild? 250) and no more bot crashing. i can't thank you enough, the help i got from you is much appreciated.

[moo]

do you have the error_log when it crashed?

[starams5]

last crash:
line 17: [Sat Jun 16 06:56:55 2012] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild? setting
[Sat Jun 16 07:02:53 2012] [notice] Parent: Received shutdown signal -- Shutting down the server.

old ThreadsPerChild? setting:
line 143: [Sat Jun 16 09:39:42 2012] [notice] Child 4960: Starting 150 worker threads.

new ThreadsPerChild? settings:
line 167: [Sat Jun 16 11:19:02 2012] [notice] Child 1204: Starting 250 worker threads.

the crashes were occuring daily, usually in the early AM hours (100% CPU). its been 17 hours since the last crash, haven't crashed since raising the ThreadsPerChild? to 250, still monitoring the situation. thanks, see attached log.

[moo]

ok. not a XCache problem. not even SIGSEGV (or any other way) the program die "crash")

[Sat Jun 16 06:56:55 2012] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild?? setting <- too many keep alive requests holding all 150 worker threads so no thread to serve the new request. and yes raising like it is suggested.
[Sat Jun 16 07:02:53 2012] [notice] Parent: Received shutdown signal -- Shutting down the server. <- stopped (restarting) by you

[starams5]

i didn't think it was an xcache problem, there was nothing in the log to indicate it was. the problem was on my end. the server/xcache has been running much smoother since raising the ThreadsPerChild? to 250. thanks moo for taking the time to check it.

[starams5]

Hi Moo,

The crashing came back, crashed a few times today. This time I checked the Windows Application log and this is what I found. Thanks.

---

Faulting application name: httpd.exe, version: 2.2.21.0, time stamp: 0x4e6b3136

Faulting module name: php_xcache.dll, version: 5.4.3.0, time stamp: 0x4fd71256

Exception code: 0xc0000005

Fault offset: 0x000138c5

Faulting process id: 0x113c

Faulting application start time: 0x01cd4e2ea143b65e

Faulting application path: C:\xampp\apache\bin\httpd.exe

Faulting module path: C:\xampp\php\ext\php_xcache.dll

Report Id: 0eeece72-ba52-11e1-847c-bcaec544343e

[moo]

do you have error log for it this time?

[starams5]

See attached log. Update to previous post. I downloaded XCache-trunk-r908-php-5.4.3-Win32-VC9-x86 and installed it. Not sure what changes were made, we'll see if the problem persist. Thanks.

[starams5]

hi moo,

I had to go back to APC for now, rebooting apache every time traffic picked up (or google bots came) was bad for business. for testing, might try the "Webserver Stress Tool". the site is slow again with APC, i'll keep an eye out for xcache updates, don't like the site being slow again its stable.

thanks

[moo]

the new r914 snapshot is uploaded to same url. i've added crash dmp generating to it, but i don't have time yet to make sure it really dump a php-xcache.dmp file in %TMP% directory. i need the dmp file

[moo]

when crash happens, php-xcache.dmp should be generated inside "xcache.coredump_directory" only if "xcache.coredump_directory" is set (not %TMP%)

[starams5]

okay, thanks. i just enabled the xcache.coredump_directory on the test site, i'll try it there first. i'll get back to you. by the way, i put everything back the way it was (PHP Version 5.3.8) to ensure all modules versions were correct (xampp). currently using XCache-trunk-r914-php-5.3.14-Win32-VC9-x86

thanks.

[moo]

updated. please download and use r916 for a stress test instead :P
XCache-trunk-r914-php-5.3.14 is compatible with all PHP 5.3.x

[starams5]

okay, i got it to crash on the test site but ran into a problem. apache error log: Fatal error</b>: Unable to enable crash dump saver: DBGHELP.DLL too old

i found DBGHELP.DLL in windows/system32, now i have to locate the right version of the .dll. current version (DBGHELP.DLL) 6.1.7601.17514

[moo]

get an updated DBGHELP.DLL which is also compatible with your system, to be aside of php_xcache.dll. you can normal find it in one of the applications you install. otherwise, download/install "debugging tools for windows", copy dbghelp.dll from it

[starams5]

okay, im indexing directories where it might be located now. i'll get back to you when i get it ironed out. thanks.

[starams5]

could you upload the DBGHELP.DLL you're using, it would save a lot of time. i took permission of the file, replaced it with the dll from the debugging tools for windows, rebooted the computer and now apache error log says file not found. my test site is installed on Windows 7 x64. thanks.

[moo]

sorry. win7 is update to date. i was busy and made mistakes. blind coding without testing is useless. the "too old" check was false positive.

is it possible for me to test the test site on your box? instructions is needed. mailto:phpxcache@…

[moo]

r918 win32 binaries is uploaded

[starams5]

thanks moo, the dump is working now. the file failed to attach so i uploaded the dump files to mediafire. i started the stress test with 150 users and apache crashed. download xcache.zip (dump files):  http://www.mediafire.com/?9cwj07zpcikx93r

[moo]

i've read the core dump. looks like it's corrupted and cause segv in 2 different way. the dump is somehow useful but it seems more info is needed. please upgrade to r921 and generate more dmp file. dmp file contains security data, so please use email instead. mailto:phpxcache@…

 	php5ts.dll!_zend_mm_free_int(_zend_mm_heap * heap=0x121884c8, void * p=0x00000000)  Line 2028	C
 	php5ts.dll!_efree(void * ptr=0x01234250)  Line 2361 + 0xa bytes	C
 	php5ts.dll!_zval_dtor_func(_zval_struct * zvalue=0x03ac0440)  Line 36 + 0xc bytes	C
 	php5ts.dll!_zval_ptr_dtor(_zval_struct * * zval_ptr=0x1c1e7b74)  Line 447 + 0xc bytes	C
 	php5ts.dll!_zend_hash_add_or_update(_hashtable * ht=0x1c1f0fb0, const char * arKey=0x6464decc, unsigned int nKeyLength=0x00000003, void * pData=0x1081e59c, unsigned int nDataSize=0x00000004, void * * pDest=0x00000000, int flag=0x00000001)  Line 237 + 0x6 bytes	C
 	php5ts.dll!zim_reflection_method___construct(int ht=0x00000002, _zval_struct * return_value=0x1c1f32c0, _zval_struct * * return_value_ptr=0x00000000, _zval_struct * this_ptr=0x1c1f3260, int return_value_used=0x00000000, void * * * tsrm_ls=0x11cb65c8)  Line 2529	C
 	php5ts.dll!zend_do_fcall_common_helper_SPEC(_zend_execute_data * execute_data=0x1158d488, void * * * tsrm_ls=0x11cb6501)  Line 320 + 0x41 bytes	C
>	php5ts.dll!ZEND_DO_FCALL_BY_NAME_SPEC_HANDLER(_zend_execute_data * execute_data=0x00000005, void * * * tsrm_ls=0x08642558)  Line 426	C

>	php5ts.dll!_zend_mm_free_int(_zend_mm_heap * heap=0x27fcf578, void * p=0x00000000)  Line 2029 + 0xad bytes	C
 	php5ts.dll!_efree(void * ptr=0x04fb98d8)  Line 2361 + 0xa bytes	C
 	php5ts.dll!destroy_op_array(_zend_op_array * op_array=0x04fb98e0, void * * * tsrm_ls=0x3140fa48)  Line 239 + 0x9 bytes	C
 	php5ts.dll!zend_function_dtor(_zend_function * function=0x26496d08)  Line 127 + 0x16 bytes	C
 	php5ts.dll!zend_hash_destroy(_hashtable * ht=0x3864fc9c)  Line 529 + 0x6 bytes	C
 	php5ts.dll!destroy_zend_class(_zend_class_entry * * pce=0x324d7ebc)  Line 192	C
 	php5ts.dll!zend_hash_apply_deleter(_hashtable * ht=0x00000000, bucket * p=0x04fb98d0)  Line 612 + 0x6 bytes	C

[starams5]

i deleted the files from mediafire. i sent you the dump files via email. i'll run more test with less users, maybe that will prevent corruption. thanks.

[starams5]

check email, i sent 3 more zip files. thanks.

[moo]

OK. the latter issue is a PHP 5.3.13 issue which is fixed in PHP side.