compiling yes after some time

[CyberCr33p]

On one of my servers the xcache after some time shows for php#0 or php#1 compiling = yes and after this I see a lot of clogs.

I tried to disable ioncube, change php-fpm from dynamic php processes to static processes and still the same issue. The php processes don't crash (no SIGSEGV) as I can see if crash (SIGSEGV) from php-fpm.log when I use log level = debug.

I tried to store core dumps on /tmp/phpcore but nothing there either as it doesn't core dump.

Do you have any tips to troubleshoot it or any idea what it may cause it?

[moo]

is there anything would kill php-fpm by using SIGKILL instead of SIGTERM? or will php-fpm kill itself?

[CyberCr33p]

No nothing kills php-fpm. The same php-fpm keep running.

[CyberCr33p]

Also I enable xcache.readonly_protection and change xcache.mmap_path to /tmp/xcache and issue still happens.

[moo]

open xcache.c, and remove the following code

    if (cache->compiling) {
        cache->clogs ++;
        return old_compile_file(h, type TSRMLS_CC);
    }

recompile/reinstall it and see what happen

[CyberCr33p]

I remove these lines and compile xcache again but same issue happens again.

Any other idea?

[moo]

no, it should not. it may be in compiling status but next compile request should come in and override it if success

[CyberCr33p]

Here is what I did:

I use the freebsd ports system to compile it.

1) I download xcache-1.3.0.tar.bz2
2) I uncompress the file
3) I remove the lines you told me from xcache.c
4) I compress the file and store it on /usr/ports/distfiles
5) I change the distinfo file from freebsd xcache port to contain the new values for MD5, SHA256 and SIZE
6) I did make install from the ports directory
7) When I go to the work directory (/usr/ports/www/xcache/work/xcache-1.3.0) I can see the xcache.c file with the removed lines

[CyberCr33p]

I contact #freebsd on freenode and they told me an easier way to compile it but the method I use works too.

[CyberCr33p]

I am not a programmer so I have basic knownledge but I believe that after the cache crash the cache->compiling value remains the same for the time that PHP process is running.

I found this code:

                if (cache->compiling) {
                        cache->clogs ++;
                        op_array = NULL;
                        clogged = 1;
                        break;
                }

and few lines after this code:

cache->compiling = XG(request_time);

and few lines after this code:

        if (clogged) {
                return old_compile_file(h, type TSRMLS_CC);
        }

Maybe the problem with this is the line with the break; ? Because if it breaks then the function stops and it doesn't give a new value to cache->compiling

As I said I am not a programmer so I may be completely wrong.

[CyberCr33p]

I remove:

        if (cache->compiling) {
                cache->clogs ++; /* is it safe here? */
                return old_compile_file(h, type TSRMLS_CC);
        }

and

                if (cache->compiling) {
                        cache->clogs ++;
                        op_array = NULL;
                        clogged = 1;
                        break;
                }

and now xcache runs for the last 4 hours without hanging on Compiling.

I will keep running it for more hours to see if these changes fix the issue.

[CyberCr33p]

The same result even after the changes I made. Any other idea about this?

[moo]

bad news, looks like cache corruption. have to review PHP 5.3 code changes to fix it

[CyberCr33p]

If you want me to test something let me know. Thanks for the help.

[CyberCr33p]

I try to find why the clogs keeping increasing as I have delete the lines that increase it ( cache->clogs ++; )

Any idea why ?

[moo]

are you sure it's increasing? or is it a random number?

if it's increasing, you're overwriting wrong xcache.so or haven't restart php yet?

[CyberCr33p]

Don't know if it's random number. At every refresh it's different. I will check it again and reply to your question in few hours.

[CyberCr33p]

More information:

I load xcache using php extensions and not zend extensions and same issue.

[CyberCr33p]

OK I found what I did wrong on my first comments about disabling:

        if (cache->compiling) {
                cache->clogs ++; /* is it safe here? */
                return old_compile_file(h, type TSRMLS_CC);
        }

I had change it to

/*
        if (cache->compiling) {
                cache->clogs ++; /* is it safe here? */
                return old_compile_file(h, type TSRMLS_CC);
        }
*/

which conflicts with the comment " is it safe here? "

Now I completely remove

        if (cache->compiling) {
                cache->clogs ++; /* is it safe here? */
                return old_compile_file(h, type TSRMLS_CC);
        }

and also remove

                if (cache->compiling) {
                        cache->clogs ++;
                        op_array = NULL;
                        clogged = 1;
                        break;
                }

and now the clogs stay 0

I will post a new reply tomorrow with the result.

[CyberCr33p]

The same issue again. Compiling to "Yes". After the changes I made to my previous post, I see no clogs increasing.

Is any way to add a counter and after some clogs, for example 100 clogs to reset the cache?

[CyberCr33p]

OK some more information. It shows clogs 0 and the Hits are increasing. But Compiling stays to "Yes". I believe now with these 2 changes it uses the cache even if there is a problem with the cache->compiling.

Let me know what you think about my idea on the previous post.

[moo]

did you remove or comment out cache->compiling = XG(request_time); on the last retry? this should be enough to keep compiling to "no" and other changes is not needed. unless you still get compiling as "yes".

in case if you still get yes, change any "if (cache->compiling) {" to "if (false && cache->compiling) {", it's still yes but cache should be working. we'll see why it's yes

since it's "working", keep it as it for now, let's run it for a while see if the cache get corrupted

[CyberCr33p]

I remove

cache->compiling = XG(request_time);

without any other changes to xcache.c

and now it runs stable for the last 9+ hours.

[CyberCr33p]

After 16 hours running xcache I can say that removing this line fix the issue.

But what is the downside of this removal?

[moo]

you lost the ability for 1 compiler writer + multiple reader concurrent access to the cache

you need it for high concurrent and frequent cache update (for new php files), i'll see why the compiling flag is set and not clear

[CyberCr33p]

The same issue happens on other server too. It happens after few days on this server but the PHP usage on this server is less.

[CyberCr33p]

I upgrade to 1.3.2-rc1 to see if it resolve this issue.

[CyberCr33p]

I run it for 20 hours and it looks stable. Did you made anything changes on version 1.3.2 regarding this issue?

[CyberCr33p]

After running it for 4 days I can safely say that it solve this issue.

[moo]

thanks for your confirmation