I have this recurring issue where an httpd processes will randomly start running at 100% CPU. Often other httpd processes will join in, and it will continue until I restart Apache. Oddly, the thing it's running 100% CPU on is "OPTIONS * HTTP/1.0". Here's the output from one of these:
9-0 38787 1/9/391 C 103.14 1323 7 0.0 0.08 4.11 ::1 www.mysite.com OPTIONS * HTTP/1.0
CPU is at 103.14% and it's been 1323 seconds since its last request. It's also stuck in 'C' - closing connection state.
Here's another case where other processes join in running 100% CPU:
0-0 12792 0/33/64 W 95.73 1097 0 0.0 0.10 0.39 66.68.237.216 www.mysite.com POST /page_a.php HTTP/1.1
9-0 12795 1/6/15 C 94.42 1174 0 0.0 0.03 0.07 ::1 www.myserver.com OPTIONS * HTTP/1.0
19-0 12986 0/4/41 W 95.67 1011 0 0.0 0.03 0.24 81.237.216.111 www.mysite.com POST /page_b.php HTTP/1.1
20-0 12720 0/10/10 W 94.32 1220 0 0.0 0.03 0.03 187.184.103.218 www.mysite.com POST /page_a.php HTTP/1.1
My setup is that this is on OS X Lion 10.4.7. I'm running Apache2 with PHP 5.3. Server Version: Apache/2.2.21 (Unix) mod_ssl/2.2.21 OpenSSL/0.9.8r DAV/2 PHP/5.3.10 with Suhosin-Patch
Some potentially relevant http.conf settings:
MaxRequestsPerChild 100000
Timeout 300
KeepAlive On
KeepAliveTimeout 8
MaxKeepAliveRequests 100
When I sample the runaway processes, I see each of them show _spin_lock and kevent like so:
Sort by top of stack, same collapsed (when >= 5):
_spin_lock$VARIANT$mp (in libsystem_c.dylib) 4358
kevent (in libsystem_kernel.dylib) 2179
Here is some of the non-repeating Apple code I see:
Call graph:
2413 Thread_1080146 DispatchQueue_1: com.apple.main-thread (serial)
+ 2413 start (in httpd) + 52 [0x10f364794]
+ 2413 main (in httpd) + 4045 [0x10f37048d]
+ 2413 ap_mpm_run (in httpd) + 1740 [0x10f3aaabc]
+ 2413 perform_idle_server_maintenance (in httpd) + 703 [0x10f3aa38f]
+ 2413 make_child (in httpd) + 435 [0x10f3aa003]
+ 2413 child_main (in httpd) + 1831 [0x10f3a9e47]
+ 2413 clean_child_exit (in httpd) + 49 [0x10f3a8d31]
+ 2413 apr_pool_destroy (in libapr-1.0.dylib) + 52 [0x10f45761b]
+ 2413 ??? (in libapr-1.0.dylib) load address 0x10f44b000 + 0xb7b4 [0x10f4567b4]
+ 2413 php_apache_child_shutdown (in libphp5.so) + 17 [0x10fa3842f]
+ 2413 php_module_shutdown_wrapper (in libphp5.so) + 9 [0x10f97021d]
+ 2413 php_module_shutdown (in libphp5.so) + 35 [0x10f970167]
+ 2413 zend_shutdown (in libphp5.so) + 57 [0x10f9c0692]
+ 2413 zend_hash_destroy (in libphp5.so) + 53 [0x10f9cb19a]
+ 2413 destroy_op_array (in libphp5.so) + 271 [0x10f9b91c0]
+ 2413 _efree (in libphp5.so) + 52 [0x10f9a6312]
+ 2413 _zend_mm_free_canary_int (in libphp5.so) + 473 [0x10f9db899]
+ 2413 free (in libsystem_c.dylib) + 71 [0x7fff8f86170e]
+ 2413 szone_size_try_large (in libsystem_c.dylib) + 37 [0x7fff8f8240f9]
+ 2413 _spin_lock$VARIANT$mp (in libsystem_c.dylib) + 30,25,... [0x7fff8f86336e,0x7fff8f863369,...]
2413 Thread_1080153 DispatchQueue_2: com.apple.libdispatch-manager (serial)
2413 _dispatch_mgr_thread (in libdispatch.dylib) + 54 [0x7fff908fc31a]
2413 _dispatch_mgr_invoke (in libdispatch.dylib) + 923 [0x7fff908fd78a]
2413 kevent (in libsystem_kernel.dylib) + 10 [0x7fff8ed047e6]
And then stuff like this:
0x10f4ce000 - 0x10f4d0ff7 mod_reqtimeout.so (??? - ???) <035F872B-8196-3CCE-A4D0-AA8D5C1550EC> /usr/libexec/apache2/mod_reqtimeout.so
0x10f4d4000 - 0x10f4d8ff7 mod_ext_filter.so (??? - ???) <F33A3409-BFBA-3C8F-9FC9-C1BBC17DBE3F> /usr/libexec/apache2/mod_ext_filter.so
0x10f4dd000 - 0x10f4eaff7 mod_include.so (??? - ???) <70E541B9-A864-3FE1-AB85-EBF632FFD376> /usr/libexec/apache2/mod_include.so
0x10f4ef000 - 0x10f4f2ff7 mod_filter.so (??? - ???) <2093EE45-E335-3B36-A6BA-6EA4EB7E483C> /usr/libexec/apache2/mod_filter.so
0x10f4f6000 - 0x10f4f8ff7 mod_substitute.so (??? - ???) <9ED1AB37-EE13-39DC-AB97-98A2B39555B0> /usr/libexec/apache2/mod_substitute.so
0x10f4fc000 - 0x10f501ff7 mod_deflate.so (??? - ???) <E334FEA5-BB4E-36BE-A364-DCE248793345> /usr/libexec/apache2/mod_deflate.so
0x10f506000 - 0x10f50bfff mod_log_config.so (??? - ???) <61EA3051-8D4A-3A00-B7BC-C68E18CA9479> /usr/libexec/apache2/mod_log_config.so
0x10f511000 - 0x10f512fef mod_log_forensic.so (??? - ???) <06654BB4-CA2A-3D70-B759-12191119E5C7> /usr/libexec/apache2/mod_log_forensic.so
0x10f516000 - 0x10f516ff7 mod_logio.so (??? - ???) <A3CE3AED-D5FE-3947-8CEB-0A8F4C534E86> /usr/libexec/apache2/mod_logio.so
0x10f51a000 - 0x10f51aff7 mod_env.so (??? - ???) <DE58A08A-C6C9-307B-A077-03A9F1AA14C5> /usr/libexec/apache2/mod_env.so
0x10f51e000 - 0x10f524fff mod_mime_magic.so (??? - ???) <1737F398-6315-31B4-B8B4-57F590F07268> /usr/libexec/apache2/mod_mime_magic.so
0x10f529000 - 0x10f52aff7 mod_cern_meta.so (??? - ???) <B0D4FF96-C96D-3F3A-AECD-431F22A998FA> /usr/libexec/apache2/mod_cern_meta.so
0x10f52e000 - 0x10f52ffff mod_expires.so (??? - ???) <FE97E6F0-3B84-3CAC-8873-3B1525142041> /usr/libexec/apache2/mod_expires.so
0x10f533000 - 0x10f536ff7 mod_headers.so (??? - ???) <5701D330-D777-3AAF-AEEF-F02D067F851E> /usr/libexec/apache2/mod_headers.so
0x10f53b000 - 0x10f53cfff mod_ident.so (??? - ???) <5FDFBB79-3A0C-3439-BF71-74E6A3A4B7AC> /usr/libexec/apache2/mod_ident.so
0x10f540000 - 0x10f542ff7 mod_usertrack.so (??? - ???) <09F36BB5-4F8D-339B-AA52-5FA23A946837> /usr/libexec/apache2/mod_usertrack.so
0x10f546000 - 0x10f547fff mod_setenvif.so (??? - ???) <D312073B-0320-3810-9535-3AB8EBF83659> /usr/libexec/apache2/mod_setenvif.so
0x10f54b000 - 0x10f54cff7 mod_version.so (??? - ???) <4BF2E21C-E452-340E-A6B4-196DBD731CA8> /usr/libexec/apache2/mod_version.so
0x10f550000 - 0x10f566fff mod_proxy.so (??? - ???) <0169F3B2-A81A-3E23-92FE-8E9B92C38795> /usr/libexec/apache2/mod_proxy.so
0x10f56e000 - 0x10f576ff7 mod_proxy_http.so (??? - ???) <F5B55152-D3D1-34AE-A95A-029F01476A54> /usr/libexec/apache2/mod_proxy_http.so
0x10f57c000 - 0x10f57efff mod_proxy_scgi.so (??? - ???) <DFD46BF1-F69D-383B-8B33-5273DD8037E4> /usr/libexec/apache2/mod_proxy_scgi.so
0x10f583000 - 0x10f589ff7 mod_proxy_balancer.so (??? - ???) <BC9C283D-8852-3016-8142-96A5675956FA> /usr/libexec/apache2/mod_proxy_balancer.so
I can post more details as requested.
