Opened 2 years ago

Closed 2 years ago

Last modified 21 months ago

#281 closed defect (fixed)

Apache Crashes Under Load With XCache

Reported by: starams5 Owned by: moo
Priority: major Milestone: 3.0.1
Component: cacher Version: 2.0.0
Keywords: Cc:
Application: PHP Version: 5.3.8
Other Exts: SAPI: apache2handler
Probability: Sometimes Blocked By:
Blocking:

Description

First, I want to say you guys did a good job with XCache 2.0.0 Final (XCache-2.0.0-php-5.3.10-Win32-VC9-x86). This version is more stable than the 1xx versions. I only have one problem, when the server starts to get under load (80+ users), Apache crashes and the Apache service will stops. This doesn't happen with APC or file based.

I've tried different settings, but I'm no export with these settings. Could you please help, and explain in detail what I can change to get this working.

Specs:
Windows Server 2008 R2
Duel Quad Core Xeons
16Gb Memory

XAMPP 1.7.7, which includes:

Apache 2.2.21
MySQL 5.5.16
PHP 5.3.8 (Memory Limit Set At 128M)

Thanks

Attachments (8)

error.log (15.6 KB) - added by starams5 2 years ago.
Apache Error Log
Region Capture.png (67.6 KB) - added by starams5 2 years ago.
XCache Admin Image
XCache-2.0.0-php-5.4.0-Win32-VC9-x86.png (15.0 KB) - added by starams5 2 years ago.
XCache-trunk-r903-php-5.4.3-nts-Win32-VC9-x86.png (14.9 KB) - added by starams5 2 years ago.
Error.png (49.4 KB) - added by starams5 2 years ago.
xcache admin.png (9.1 KB) - added by starams5 2 years ago.
error.2.log (41.3 KB) - added by starams5 2 years ago.
error.3.log (11.9 KB) - added by starams5 2 years ago.

Download all attachments as: .zip

Change History (69)

comment:1 Changed 2 years ago by moo

is your cache almost full before the cash? it is reported in #246 and is fixed. i'm gonna role out a release soon

comment:2 Changed 2 years ago by starams5

thanks. i doubt the cache's were full. i tried 60M-512M, the results were always the same, Apache crash. it was hard going back to ACP after using XCache. the difference is noticeable. with XCache the site was faster & smoother until site trafic became heavy. i'll just wait for the next release, hopefully the problem is resolved.

thanks.

comment:3 Changed 2 years ago by starams5

not sure if this will help or not. i uploaded an image of the Apache error log, see link:

http://img545.imageshack.us/img545/7043/regioncapture.png

comment:4 Changed 2 years ago by moo

please attach image here using "attach file". for error_log you can post text here directly

comment:5 Changed 2 years ago by starams5

i've been testing on my test site using stress software, and the crashes continued (80+ users). then i tried the settings below, and so far no more crashes. before i move XCache to my live site, could you give your opinion on the settings below (all other settings default):

; to enable : xcache.size=64M etc (any size > 0) and your system mmap allows
xcache.size = 0M

comment:6 Changed 2 years ago by moo

xcache.size = 0M means no cache. i'll try to stress it too but i wonder what php code can cause this issue. cause i sure don't have the same code as you do

comment:7 Changed 2 years ago by starams5

Update. We may have solved the problem. I updated to PHP 5.4 (& XCache PHP 5.4 version). Updated the necessary .dll's, and took care of the errors in the Apache log. Tested on the test site with 140 users, 15 clicks per user, 20 second delay per user. No Apache crashes.

Moved everything to the live site, all of the above, no errors. If you could help me with the settings, I would appreciate it. I have a large site with a lot of files loading, and I think it should be a little faster than it is, the settings are probably wrong. Could you please recommend optimum settings. Thanks.

Current configuration:

[xcache-common]
;; install as zend extension (recommended), normally "$extension_dir/xcache.so"
;zend_extension = /usr/local/lib/php/extensions/non-debug-non-zts-xxx/xcache.so
; zend_extension_ts = /usr/local/lib/php/extensions/non-debug-zts-xxx/xcache.so
;; For windows users, replace xcache.so with php_xcache.dll
zend_extension_ts = c:/xampp/php/ext/php_xcache.dll
;; or install as extension, make sure your extension_dir setting is correct
; extension = xcache.so
;; or win32:
extension = php_xcache.dll

[xcache.admin]
xcache.admin.enable_auth = Off
xcache.admin.user = "runner"
; xcache.admin.pass = md5($your_password)
xcache.admin.pass = "5f4dcc3b5aa765d61d8327deb882cf99"

[xcache]
; ini only settings, all the values here is default unless explained

; select low level shm/allocator scheme implemenation
xcache.shm_scheme = "mmap"
; to disable: xcache.size=0
; to enable : xcache.size=64M etc (any size > 0) and your system mmap allows
xcache.size = 128M
; set to cpu count (cat /proc/cpuinfo |grep -c processor)
xcache.count = 1
; just a hash hints, you can always store count(items) > slots
xcache.slots = 8k
; ttl of the cache item, 0=forever
xcache.ttl = 300
; interval of gc scanning expired items, 0=no scan, other values is in seconds
xcache.gc_interval = 480

; same as aboves but for variable cache
xcache.var_size = 4M
xcache.var_count = 1
xcache.var_slots = 8K
; default ttl
xcache.var_ttl = 0
xcache.var_maxttl = 0
xcache.var_gc_interval = 300

; N/A for /dev/zero
xcache.readonly_protection = Off
; for *nix, xcache.mmap_path is a file path, not directory.
; Use something like "/tmp/xcache" if you want to turn on ReadonlyProtection
; 2 group of php won't share the same /tmp/xcache
; for win32, xcache.mmap_path=anonymous map name, not file path
xcache.mmap_path = "/dev/zero"

; leave it blank(disabled) or "/tmp/phpcore/"
; make sure it's writable by php (open_basedir is not checked)
xcache.coredump_directory = ""

; per request settings
xcache.cacher = On
xcache.stat = On
xcache.optimizer = Off

xcache.test = Off
xcache.experimental = Off

[xcache.coverager]
; per request settings
; enable coverage data collecting for xcache.coveragedump_directory and xcache_coverager_start/stop/get/clean() functions (will hurt executing performance)
xcache.coverager = Off

; ini only settings
; make sure it's readable (open_basedir is checked) by coverage viewer script
; requires xcache.coverager=On
xcache.coveragedump_directory = ""

comment:8 Changed 2 years ago by moo

it seems "zend_extension_ts = c:/xampp/php/ext/php_xcache.dll" should be "zend_extension = c:/xampp/php/ext/php_xcache.dll" since PHP 5.4. why not put a phpinfo() page and see if xcache is loaded. and xcache.size = 128M may not enough if you have lots of php files. just put xcache/admin/* to your htdocs/xcache-admin/ to see how your cache is used

Changed 2 years ago by starams5

Apache Error Log

comment:9 Changed 2 years ago by moo

what url do you request to get this error triggered so it is logged. is it the 1st time request after you start apache or the 2nd time? if it's easy to reproduce could you please make an short reproduce-able code?

comment:10 Changed 2 years ago by moo

  • PHP Version changed from 5.3.8 to 5.4.3

comment:11 Changed 2 years ago by starams5

thanks for your help. I have a media site, videos, audio, etc; I publish the updates to Facebook, Twittter, & a few more networks. This is when Apache crashes, when the people start coming to the site.

I'm working on the xcache admin now. .htaccess is not allowing access to xcache admin, trying to find a work-around.

Thanks.

Changed 2 years ago by starams5

XCache Admin Image

comment:12 Changed 2 years ago by starams5

Okay, I got into the admin, please see attached file.

comment:13 Changed 2 years ago by moo

for now you may have to disable XCache. within a few days, i'm gonna add mini dump support for PHP with XCache enabled to generate php.dmp before i can analyze the crash

comment:14 Changed 2 years ago by moo

maybe fixed in [903]. i'll roll out an win32 build soon for your testing before i close this ticket.

comment:15 Changed 2 years ago by starams5

Okay thanks. I'm still using xcache, but watching the situation closely. Although the errors appear in the log, its staying online for now. Its hard to go back to APC after using xcache. My site is slow with APC. I'll keep checking the site for the new build. Thanks.

comment:16 Changed 2 years ago by moo

  • Resolution set to fixed
  • Status changed from new to closed

please download from latest snapshot from http://xcache.lighttpd.net/pub/snapshots/ for your favorite version. reopen this ticket if it's not fixed

comment:17 Changed 2 years ago by moo

btw, if things go good. report. if things go bad, attach new error.log again. thanks for your helping to make XCache better

comment:18 Changed 2 years ago by starams5

i sure will. either way, you're the best, man. thanks for all your help. i'll test when things slow down a bit. thanks.

Changed 2 years ago by starams5

comment:19 Changed 2 years ago by starams5

I'm not sure what's going on, but my caching option will not except XCache-trunk-r903-php-5.4.3-nts-Win32-VC9-x86 (option to enable is grayed out). See attached screen-shots. Thanks.

Test site configuration:

[xcache-common]
;; install as zend extension (recommended), normally "$extension_dir/xcache.so"
;zend_extension = /usr/local/lib/php/extensions/non-debug-non-zts-xxx/xcache.so
; zend_extension_ts = /usr/local/lib/php/extensions/non-debug-zts-xxx/xcache.so
;; For windows users, replace xcache.so with php_xcache.dll
zend_extension_ts = f:/xampp/php/ext/php_xcache.dll
;; or install as extension, make sure your extension_dir setting is correct
; extension = xcache.so
;; or win32:
extension = php_xcache.dll

[xcache.admin]
xcache.admin.enable_auth = Off
xcache.admin.user = "runner"
; xcache.admin.pass = md5($your_password)
xcache.admin.pass = "5f4dcc3b5aa765d61d8327deb882cf99"

[xcache]
; ini only settings, all the values here is default unless explained

; select low level shm/allocator scheme implemenation
xcache.shm_scheme = "mmap"
; to disable: xcache.size=0
; to enable : xcache.size=64M etc (any size > 0) and your system mmap allows
xcache.size = 60M
; set to cpu count (cat /proc/cpuinfo |grep -c processor)
xcache.count = 1
; just a hash hints, you can always store count(items) > slots
xcache.slots = 8K
; ttl of the cache item, 0=forever
xcache.ttl = 0
; interval of gc scanning expired items, 0=no scan, other values is in seconds
xcache.gc_interval = 0

; same as aboves but for variable cache
xcache.var_size = 4M
xcache.var_count = 1
xcache.var_slots = 8K
; default ttl
xcache.var_ttl = 0
xcache.var_maxttl = 0
xcache.var_gc_interval = 300

; N/A for /dev/zero
xcache.readonly_protection = Off
; for *nix, xcache.mmap_path is a file path, not directory.
; Use something like "/tmp/xcache" if you want to turn on ReadonlyProtection
; 2 group of php won't share the same /tmp/xcache
; for win32, xcache.mmap_path=anonymous map name, not file path
xcache.mmap_path = "/dev/zero"

; leave it blank(disabled) or "/tmp/phpcore/"
; make sure it's writable by php (open_basedir is not checked)
xcache.coredump_directory = ""

; per request settings
xcache.cacher = On
xcache.stat = On
xcache.optimizer = Off

xcache.test = Off
xcache.experimental = Off

[xcache.coverager]
; per request settings
; enable coverage data collecting for xcache.coveragedump_directory and xcache_coverager_start/stop/get/clean() functions (will hurt executing performance)
xcache.coverager = Off

; ini only settings
; make sure it's readable (open_basedir is checked) by coverage viewer script
; requires xcache.coverager=On
xcache.coveragedump_directory = ""

comment:20 Changed 2 years ago by moo

it may not load correctly. check phpinfo and/or xcache-admin page again

comment:21 Changed 2 years ago by starams5

i've gone over the configuration several times, it works just fine with XCache-2.0.0-php-5.4.0-Win32-VC9-x86.

i tried swapping out the .dll with XCache-trunk-r903-php-5.4.3-nts-Win32-VC9-x86 after enabling XCache-2.0.0-php-5.4.0-Win32-VC9-x86 (re-started Apache), no luck. cant enter xcache admin, and the test site crashes. xcache admin work just fine with XCache-2.0.0-php-5.4.0-Win32-VC9-x86

see attachments. thanks.

Changed 2 years ago by starams5

Changed 2 years ago by starams5

comment:22 Changed 2 years ago by moo

hrm... looks like i failed to build the ts version (the name without nts)

comment:23 Changed 2 years ago by moo

do you have any IM so we can interact with each other in order to reduce time

comment:24 Changed 2 years ago by starams5

thanks. i don't have any IM accounts. i just re-enabled the site chat bar, we can use that unless you have another solution. only problem is, you have to log into the site to use the bar. you can log in with Facebook or Twitter accounts. i'll add you as a friend, and we can chat. thanks.

http://www.urbanepeeps.com/

comment:25 Changed 2 years ago by starams5

i took care of everything, we can chat on the bar now. account activated. thanks.

comment:26 Changed 2 years ago by moo

cannot login. no error prompted

comment:27 Changed 2 years ago by moo

thanks for your time. i'll be available at irc.freenode.net #lighttpd 5 hours later from now. you may use http://webchat.freenode.net/

i had to build non-zts first

comment:28 Changed 2 years ago by starams5

okay thanks. also, i reset the password on your account at the site, and tested the login. it's been mailed to you. thanks.

comment:29 Changed 2 years ago by moo

moo @ freenode.net#lighttpd, zts build uploaded. please try

comment:30 Changed 2 years ago by starams5

thanks moo. do you have a link?

comment:31 Changed 2 years ago by moo

same link at http://xcache.lighttpd.net/pub/snapshots/, the ones without -nts-

comment:32 Changed 2 years ago by starams5

Hi Moo,

Apologizes for not getting back to you sooner, I still work part-time job. The new .dll worked! I will test for a while before moving it to the live site. Like I said, you're the best! No one else would have took the time to help.

Thanks

comment:33 Changed 2 years ago by moo

when you done testing, make sure to report, no matter if it's buggy or not bug free

comment:34 Changed 2 years ago by starams5

sure will. i moved it to the live site. its the only way to really test it, with live traffic. so far, so good. i'll report back tonight or in the morning. thanks again for all your help.

comment:35 Changed 2 years ago by starams5

just checking back. no xcache/apache crashes. traffic got heavy yesterday, again, no crashes. i can't thank you enough. APC was okay until traffic got heavy, then the site got slow. with xcache, you can't even tell the traffic is heavy, the site remains fast & smooth with or without a load. bye bye APC!

comment:36 Changed 2 years ago by moo

ok. thanks for your confirmation to the fix for windows zend thread safe (zts) enabled. it will be released soon as 2.0.1, which is more likely an identical version you're using now

comment:37 Changed 2 years ago by starams5

how it going moo,

just an update. xcache has been performing well. i had one problem but i think that's been resolved. the bots started crashing apache, google to be specific. i raised the ThreadsPerChild? (per apache error log) and that seemed to have resolved the issue. xcache is much faster with the higher settings (ThreadsPerChild? 250) and no more bot crashing. i can't thank you enough, the help i got from you is much appreciated.

comment:38 Changed 2 years ago by moo

do you have the error_log when it crashed?

comment:39 Changed 2 years ago by starams5

last crash:
line 17: [Sat Jun 16 06:56:55 2012] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild? setting
[Sat Jun 16 07:02:53 2012] [notice] Parent: Received shutdown signal -- Shutting down the server.

old ThreadsPerChild? setting:
line 143: [Sat Jun 16 09:39:42 2012] [notice] Child 4960: Starting 150 worker threads.

new ThreadsPerChild? settings:
line 167: [Sat Jun 16 11:19:02 2012] [notice] Child 1204: Starting 250 worker threads.

the crashes were occuring daily, usually in the early AM hours (100% CPU). its been 17 hours since the last crash, haven't crashed since raising the ThreadsPerChild? to 250, still monitoring the situation. thanks, see attached log.

Changed 2 years ago by starams5

comment:40 Changed 2 years ago by moo

ok. not a XCache problem. not even SIGSEGV (or any other way) the program die "crash")

[Sat Jun 16 06:56:55 2012] [warn] Server ran out of threads to serve requests. Consider raising the ThreadsPerChild?? setting <- too many keep alive requests holding all 150 worker threads so no thread to serve the new request. and yes raising like it is suggested.
[Sat Jun 16 07:02:53 2012] [notice] Parent: Received shutdown signal -- Shutting down the server. <- stopped (restarting) by you

comment:41 Changed 2 years ago by starams5

i didn't think it was an xcache problem, there was nothing in the log to indicate it was. the problem was on my end. the server/xcache has been running much smoother since raising the ThreadsPerChild? to 250. thanks moo for taking the time to check it.

comment:42 Changed 2 years ago by starams5

Hi Moo,

The crashing came back, crashed a few times today. This time I checked the Windows Application log and this is what I found. Thanks.


Faulting application name: httpd.exe, version: 2.2.21.0, time stamp: 0x4e6b3136

Faulting module name: php_xcache.dll, version: 5.4.3.0, time stamp: 0x4fd71256

Exception code: 0xc0000005

Fault offset: 0x000138c5

Faulting process id: 0x113c

Faulting application start time: 0x01cd4e2ea143b65e

Faulting application path: C:\xampp\apache\bin\httpd.exe

Faulting module path: C:\xampp\php\ext\php_xcache.dll

Report Id: 0eeece72-ba52-11e1-847c-bcaec544343e

comment:43 Changed 2 years ago by moo

do you have error log for it this time?

Changed 2 years ago by starams5

comment:44 Changed 2 years ago by starams5

See attached log. Update to previous post. I downloaded XCache-trunk-r908-php-5.4.3-Win32-VC9-x86 and installed it. Not sure what changes were made, we'll see if the problem persist. Thanks.

comment:45 Changed 2 years ago by starams5

hi moo,

I had to go back to APC for now, rebooting apache every time traffic picked up (or google bots came) was bad for business. for testing, might try the "Webserver Stress Tool". the site is slow again with APC, i'll keep an eye out for xcache updates, don't like the site being slow again its stable.

thanks

comment:46 Changed 2 years ago by moo

  • Resolution fixed deleted
  • Status changed from closed to reopened

the new r914 snapshot is uploaded to same url. i've added crash dmp generating to it, but i don't have time yet to make sure it really dump a php-xcache.dmp file in %TMP% directory. i need the dmp file

comment:47 Changed 2 years ago by moo

when crash happens, php-xcache.dmp should be generated inside "xcache.coredump_directory" only if "xcache.coredump_directory" is set (not %TMP%)

comment:48 Changed 2 years ago by starams5

  • PHP Version changed from 5.4.3 to 5.3.8

okay, thanks. i just enabled the xcache.coredump_directory on the test site, i'll try it there first. i'll get back to you. by the way, i put everything back the way it was (PHP Version 5.3.8) to ensure all modules versions were correct (xampp). currently using XCache-trunk-r914-php-5.3.14-Win32-VC9-x86

thanks.

comment:49 Changed 2 years ago by moo

updated. please download and use r916 for a stress test instead :P
XCache-trunk-r914-php-5.3.14 is compatible with all PHP 5.3.x

comment:50 Changed 2 years ago by starams5

okay, i got it to crash on the test site but ran into a problem. apache error log: Fatal error</b>: Unable to enable crash dump saver: DBGHELP.DLL too old

i found DBGHELP.DLL in windows/system32, now i have to locate the right version of the .dll. current version (DBGHELP.DLL) 6.1.7601.17514

comment:51 Changed 2 years ago by moo

get an updated DBGHELP.DLL which is also compatible with your system, to be aside of php_xcache.dll. you can normal find it in one of the applications you install. otherwise, download/install "debugging tools for windows", copy dbghelp.dll from it

comment:52 Changed 2 years ago by starams5

okay, im indexing directories where it might be located now. i'll get back to you when i get it ironed out. thanks.

comment:53 Changed 2 years ago by starams5

could you upload the DBGHELP.DLL you're using, it would save a lot of time. i took permission of the file, replaced it with the dll from the debugging tools for windows, rebooted the computer and now apache error log says file not found. my test site is installed on Windows 7 x64. thanks.

comment:54 Changed 2 years ago by moo

sorry. win7 is update to date. i was busy and made mistakes. blind coding without testing is useless. the "too old" check was false positive.

is it possible for me to test the test site on your box? instructions is needed. mailto:phpxcache@…

comment:55 Changed 2 years ago by moo

r918 win32 binaries is uploaded

comment:56 Changed 2 years ago by starams5

thanks moo, the dump is working now. the file failed to attach so i uploaded the dump files to mediafire. i started the stress test with 150 users and apache crashed. download xcache.zip (dump files): http://www.mediafire.com/?9cwj07zpcikx93r

comment:57 Changed 2 years ago by moo

i've read the core dump. looks like it's corrupted and cause segv in 2 different way. the dump is somehow useful but it seems more info is needed. please upgrade to r921 and generate more dmp file. dmp file contains security data, so please use email instead. mailto:phpxcache@…

 	php5ts.dll!_zend_mm_free_int(_zend_mm_heap * heap=0x121884c8, void * p=0x00000000)  Line 2028	C
 	php5ts.dll!_efree(void * ptr=0x01234250)  Line 2361 + 0xa bytes	C
 	php5ts.dll!_zval_dtor_func(_zval_struct * zvalue=0x03ac0440)  Line 36 + 0xc bytes	C
 	php5ts.dll!_zval_ptr_dtor(_zval_struct * * zval_ptr=0x1c1e7b74)  Line 447 + 0xc bytes	C
 	php5ts.dll!_zend_hash_add_or_update(_hashtable * ht=0x1c1f0fb0, const char * arKey=0x6464decc, unsigned int nKeyLength=0x00000003, void * pData=0x1081e59c, unsigned int nDataSize=0x00000004, void * * pDest=0x00000000, int flag=0x00000001)  Line 237 + 0x6 bytes	C
 	php5ts.dll!zim_reflection_method___construct(int ht=0x00000002, _zval_struct * return_value=0x1c1f32c0, _zval_struct * * return_value_ptr=0x00000000, _zval_struct * this_ptr=0x1c1f3260, int return_value_used=0x00000000, void * * * tsrm_ls=0x11cb65c8)  Line 2529	C
 	php5ts.dll!zend_do_fcall_common_helper_SPEC(_zend_execute_data * execute_data=0x1158d488, void * * * tsrm_ls=0x11cb6501)  Line 320 + 0x41 bytes	C
>	php5ts.dll!ZEND_DO_FCALL_BY_NAME_SPEC_HANDLER(_zend_execute_data * execute_data=0x00000005, void * * * tsrm_ls=0x08642558)  Line 426	C
>	php5ts.dll!_zend_mm_free_int(_zend_mm_heap * heap=0x27fcf578, void * p=0x00000000)  Line 2029 + 0xad bytes	C
 	php5ts.dll!_efree(void * ptr=0x04fb98d8)  Line 2361 + 0xa bytes	C
 	php5ts.dll!destroy_op_array(_zend_op_array * op_array=0x04fb98e0, void * * * tsrm_ls=0x3140fa48)  Line 239 + 0x9 bytes	C
 	php5ts.dll!zend_function_dtor(_zend_function * function=0x26496d08)  Line 127 + 0x16 bytes	C
 	php5ts.dll!zend_hash_destroy(_hashtable * ht=0x3864fc9c)  Line 529 + 0x6 bytes	C
 	php5ts.dll!destroy_zend_class(_zend_class_entry * * pce=0x324d7ebc)  Line 192	C
 	php5ts.dll!zend_hash_apply_deleter(_hashtable * ht=0x00000000, bucket * p=0x04fb98d0)  Line 612 + 0x6 bytes	C

comment:58 Changed 2 years ago by starams5

i deleted the files from mediafire. i sent you the dump files via email. i'll run more test with less users, maybe that will prevent corruption. thanks.

comment:59 Changed 2 years ago by starams5

check email, i sent 3 more zip files. thanks.

comment:60 Changed 2 years ago by moo

  • Resolution set to fixed
  • Status changed from reopened to closed

OK. the latter issue is a PHP 5.3.13 issue which is fixed in PHP side.

comment:61 Changed 21 months ago by moo

  • Milestone changed from undecided to 3.0.1
Note: See TracTickets for help on using tickets.