Opened 3 years ago

Closed 3 years ago

#254 closed defect (fixed)

compiling yes after some time

Reported by: CyberCr33p Owned by: moo
Priority: major Milestone: 1.3.2
Component: cacher Version: 1.3.0
Keywords: xcache, clogs Cc:
Application: PHP Version: 5.3.3
Other Exts: SAPI: FastCGI
Probability: Always Blocked By:
Blocking:

Description

On one of my servers the xcache after some time shows for php#0 or php#1 compiling = yes and after this I see a lot of clogs.

I tried to disable ioncube, change php-fpm from dynamic php processes to static processes and still the same issue. The php processes don't crash (no SIGSEGV) as I can see if crash (SIGSEGV) from php-fpm.log when I use log level = debug.

I tried to store core dumps on /tmp/phpcore but nothing there either as it doesn't core dump.

Do you have any tips to troubleshoot it or any idea what it may cause it?

Change History (29)

comment:1 Changed 3 years ago by moo

is there anything would kill php-fpm by using SIGKILL instead of SIGTERM? or will php-fpm kill itself?

comment:2 Changed 3 years ago by CyberCr33p

No nothing kills php-fpm. The same php-fpm keep running.

comment:3 Changed 3 years ago by CyberCr33p

Also I enable xcache.readonly_protection and change xcache.mmap_path to /tmp/xcache and issue still happens.

comment:4 Changed 3 years ago by moo

open xcache.c, and remove the following code

    if (cache->compiling) {
        cache->clogs ++;
        return old_compile_file(h, type TSRMLS_CC);
    }

recompile/reinstall it and see what happen

comment:5 Changed 3 years ago by CyberCr33p

I remove these lines and compile xcache again but same issue happens again.

Any other idea?

comment:6 Changed 3 years ago by moo

no, it should not. it may be in compiling status but next compile request should come in and override it if success

comment:7 Changed 3 years ago by CyberCr33p

Here is what I did:

I use the freebsd ports system to compile it.

1) I download xcache-1.3.0.tar.bz2
2) I uncompress the file
3) I remove the lines you told me from xcache.c
4) I compress the file and store it on /usr/ports/distfiles
5) I change the distinfo file from freebsd xcache port to contain the new values for MD5, SHA256 and SIZE
6) I did make install from the ports directory
7) When I go to the work directory (/usr/ports/www/xcache/work/xcache-1.3.0) I can see the xcache.c file with the removed lines

comment:8 Changed 3 years ago by CyberCr33p

I contact #freebsd on freenode and they told me an easier way to compile it but the method I use works too.

comment:9 Changed 3 years ago by CyberCr33p

I am not a programmer so I have basic knownledge but I believe that after the cache crash the cache->compiling value remains the same for the time that PHP process is running.

I found this code:

                if (cache->compiling) {
                        cache->clogs ++;
                        op_array = NULL;
                        clogged = 1;
                        break;
                }

and few lines after this code:

cache->compiling = XG(request_time);

and few lines after this code:

        if (clogged) {
                return old_compile_file(h, type TSRMLS_CC);
        }

Maybe the problem with this is the line with the break; ? Because if it breaks then the function stops and it doesn't give a new value to cache->compiling

As I said I am not a programmer so I may be completely wrong.

comment:10 Changed 3 years ago by CyberCr33p

I remove:

        if (cache->compiling) {
                cache->clogs ++; /* is it safe here? */
                return old_compile_file(h, type TSRMLS_CC);
        }

and

                if (cache->compiling) {
                        cache->clogs ++;
                        op_array = NULL;
                        clogged = 1;
                        break;
                }

and now xcache runs for the last 4 hours without hanging on Compiling.

I will keep running it for more hours to see if these changes fix the issue.

comment:11 Changed 3 years ago by CyberCr33p

The same result even after the changes I made. Any other idea about this?

comment:12 Changed 3 years ago by moo

bad news, looks like cache corruption. have to review PHP 5.3 code changes to fix it

comment:13 Changed 3 years ago by CyberCr33p

If you want me to test something let me know. Thanks for the help.

comment:14 Changed 3 years ago by CyberCr33p

I try to find why the clogs keeping increasing as I have delete the lines that increase it ( cache->clogs ++; )

Any idea why ?

comment:15 Changed 3 years ago by moo

are you sure it's increasing? or is it a random number?

if it's increasing, you're overwriting wrong xcache.so or haven't restart php yet?

comment:16 Changed 3 years ago by CyberCr33p

Don't know if it's random number. At every refresh it's different. I will check it again and reply to your question in few hours.

comment:17 Changed 3 years ago by CyberCr33p

More information:

I load xcache using php extensions and not zend extensions and same issue.

comment:18 Changed 3 years ago by CyberCr33p

OK I found what I did wrong on my first comments about disabling:

        if (cache->compiling) {
                cache->clogs ++; /* is it safe here? */
                return old_compile_file(h, type TSRMLS_CC);
        }

I had change it to

/*
        if (cache->compiling) {
                cache->clogs ++; /* is it safe here? */
                return old_compile_file(h, type TSRMLS_CC);
        }
*/

which conflicts with the comment " is it safe here? "

Now I completely remove

        if (cache->compiling) {
                cache->clogs ++; /* is it safe here? */
                return old_compile_file(h, type TSRMLS_CC);
        }

and also remove

                if (cache->compiling) {
                        cache->clogs ++;
                        op_array = NULL;
                        clogged = 1;
                        break;
                }

and now the clogs stay 0

I will post a new reply tomorrow with the result.

comment:19 Changed 3 years ago by CyberCr33p

The same issue again. Compiling to "Yes". After the changes I made to my previous post, I see no clogs increasing.

Is any way to add a counter and after some clogs, for example 100 clogs to reset the cache?

comment:20 Changed 3 years ago by CyberCr33p

OK some more information. It shows clogs 0 and the Hits are increasing. But Compiling stays to "Yes". I believe now with these 2 changes it uses the cache even if there is a problem with the cache->compiling.

Let me know what you think about my idea on the previous post.

comment:21 Changed 3 years ago by moo

did you remove or comment out cache->compiling = XG(request_time); on the last retry? this should be enough to keep compiling to "no" and other changes is not needed. unless you still get compiling as "yes".

in case if you still get yes, change any "if (cache->compiling) {" to "if (false && cache->compiling) {", it's still yes but cache should be working. we'll see why it's yes

since it's "working", keep it as it for now, let's run it for a while see if the cache get corrupted

comment:22 Changed 3 years ago by CyberCr33p

I remove

cache->compiling = XG(request_time);

without any other changes to xcache.c

and now it runs stable for the last 9+ hours.

comment:23 Changed 3 years ago by CyberCr33p

After 16 hours running xcache I can say that removing this line fix the issue.

But what is the downside of this removal?

comment:24 Changed 3 years ago by moo

you lost the ability for 1 compiler writer + multiple reader concurrent access to the cache

you need it for high concurrent and frequent cache update (for new php files), i'll see why the compiling flag is set and not clear

comment:25 Changed 3 years ago by CyberCr33p

The same issue happens on other server too. It happens after few days on this server but the PHP usage on this server is less.

comment:26 Changed 3 years ago by CyberCr33p

I upgrade to 1.3.2-rc1 to see if it resolve this issue.

comment:27 Changed 3 years ago by CyberCr33p

I run it for 20 hours and it looks stable. Did you made anything changes on version 1.3.2 regarding this issue?

comment:28 Changed 3 years ago by CyberCr33p

After running it for 4 days I can safely say that it solve this issue.

comment:29 Changed 3 years ago by moo

  • Milestone set to 1.3.2
  • Resolution set to fixed
  • Status changed from new to closed

thanks for your confirmation

Note: See TracTickets for help on using tickets.