XCache dosn't cache variables when xcache.var_size >= 2048M

[mr_bond]

Problem description

Hi,

I have a problem when I set xcache.var_size to 2048M or more. What happens is php loads fine, but xcache dosn't store keys in the cache. No error is thrown when trying to receive the key from cache either. When loading the xcache-admin page, the php-process dies (FCGI).

When setting xcache.var_size to 2047M or less, everything works as expected. I tried to explore various OS limitations, but I can't find any. A process is allowed to allocate a maximum of 32GB memory, cannot find any other related hard/soft-limits, or any mmap sysctls. I noticed xcache is not failing on mmap() with ENOMEM in backtrace, but backtrace is only generated when viewing admin page. Any help/advice is much appreciated.

Environment

Xcache settings:
xcache.admin.user = "admin"
xcache.admin.pass = "ffffffffffffffffffffffffffffffff"
xcache.size = 256M
xcache.count = 4
;xcache.coredump_directory = "/tmp/core"
;xcache.var_size = 2048M
xcache.var_size = 2047M
xcache.var_count = 4
xcache.mmap_path =    "/dev/zero"

The system is running:

FreeBSD 7.0-RELEASE/amd64
PHP 5.2.12 with Suhosin-Patch 0.9.7
Xcache 1.3.0
Mem: 3051M Active, 20G Inact, 401M Wired, 446M Cache, 214M Buf, 5262M Free

Backtrace

Backtrace from admin page (does not dump core when using var_size >= 2048M):
(gdb) bt
#0  0x00000008046b76e2 in xcache_admin_operate () from /usr/local/lib/php/20060613/xcache.so
#1  0x000000000052d8e7 in zend_do_fcall_common_helper_SPEC ()
#2  0x0000000000520463 in execute ()
#3  0x000000000052449a in ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER ()
#4  0x0000000000520463 in execute ()
#5  0x0000000000501a74 in zend_execute_scripts ()
#6  0x00000000004c02f1 in php_execute_script ()
#7  0x00000000005845a9 in main ()
(gdb) dump_bt
Missing argument 0 in user function.
(gdb) dump_bt $eg.current_execute_data
Attempt to extract a component of a value that is not a structure.

[mr_bond]

I tried the patch in the ticket: http://xcache.lighttpd.net/ticket/224 but this didn't sort out the issue. I patched xcache-1.3.0_1.

What I did was to copy the port to my homedir:

# rsync -a /usr/ports/www/xcache ~/ports/www/
# cd ~/ports/www/xcache
# svn diff -r 672:675 svn://svn.lighttpd.net/xcache/trunk > files/patch-2G_limit
# make patch # (and manually check source is patched)
# make && make deinstall && make install

and then restart lighttpd.

If I set "xcache.var_size" to "3072M" xcache-admin page gives me 500 Internal server error. If I set to "4096M" only the "php cache" shows on the xcache-admin page, and it appears there is not variable caching at all. If I set it to "2048M" I get "Interal server error", if I set it back to "2047M" it works.

I'm not going to use time getting a stack-trace until somebody actually replies to the ticket. mOo?! Are you there?! :-)

[moo]

"svn diff -r 672:675 ​svn://svn.lighttpd.net/xcache/trunk" is more than what you want

try "(svn diff -c 672 ​svn://svn.lighttpd.net/xcache/trunk && svn diff -c 675 ​svn://svn.lighttpd.net/xcache/trunk) > files/patch-2G_limit"

or "svn diff -c 676 ​svn://svn.lighttpd.net/xcache/branches/1.3 > files/patch-2G_limit"

[mr_bond]

I tried "svn diff -c 676 ​svn://svn.lighttpd.net/xcache/branches/1.3 > files/patch-2G_limit":

db ~/ports/www/xcache> php -v
/libexec/ld-elf.so.1: /usr/local/lib/php/20060613/xcache.so: Undefined symbol "zend_atol"

db ~/ports/www/xcache> ldd /usr/local/lib/php/20060613/xcache.so                                                                             
/usr/local/lib/php/20060613/xcache.so:
	libc.so.7 => /lib/libc.so.7 (0x800637000)

Is this fixed in the 1.3-branch? I'll try to checkout the 1.3-branch and build it, and report back.

[moo]

back porting changes is not easy, is it?

zend_atol is defined in utils.{c,h}, but the changeset that adds it contains other changes

[671]

[mr_bond]

Hmm. I got the extra patch from [671], but it's not applying cleanly:

db ~/ports/www/xcache> make clean
db ~/ports/www/xcache> svn diff -r 669:671  svn://svn.lighttpd.net/xcache/trunk > files/patch-add_zend_atol
db ~/ports/www/xcache> make patch
===>  License check disabled, port has not defined LICENSE
===>  Found saved configuration for xcache-1.3.0_1
===>  Extracting for xcache-1.3.0_1
=> MD5 Checksum OK for xcache-1.3.0.tar.bz2.
=> SHA256 Checksum OK for xcache-1.3.0.tar.bz2.
===>  Patching for xcache-1.3.0_1
===>  Applying FreeBSD patches for xcache-1.3.0_1
Ignoring previously applied (or reversed) patch.
1 out of 1 hunks ignored--saving rejects to xcache.c.rej
Ignoring previously applied (or reversed) patch.
1 out of 1 hunks ignored--saving rejects to xcache.h.rej
=> Patch patch-add_zend_atol failed to apply cleanly.
=> Patch(es) patch-2G_limit applied cleanly.
*** Error code 1

I tried to checkout revision 676 and build from that. I ended up with 2.0.0-dev, but it works fine with >4G variable cache. However, I don't feel very confident about how well this code is tested, and If I have a mix of stable and new code from trunk.

What would be ideal is to be able to patch 1.3, or backport the changes so one can checkout the 1.3-branch with changes, or perhaps a 1.3-bigmem-branch ? What do you think?

[moo]

big ram fix in trunk was already back ported to branches/1.3

it was difficult for you to back port patch from branches/1.3 or trunk to your local 1.3.0 version

[mr_bond]

It was difficult for me to port it back to the packaged version of 1.3 in FreeBSD. It would be nice to just add the patch, and send it to maintainer. Then the patch could be incorporated into freebsd-ports before the next release. I think for debian-lenny I have to custom-build the package anyway.

I tried a checkout of the 1.3-branch, and this works with >2GB without modifications/patching it. The solution will be to build this from svn until there is a 1.3.1 release or so.

Thanks alot for your help, and effort in this project!

[moo]

okay, sounds like we have a reason to release a new version :)

[mr_bond]

That would be great :-) We are running 1.3-branch in our development-environment now - I'll report back if we run into any issues. If there is anything else I can help with, let me know.