~pcwiz Posted January 1, 2008 Share Posted January 1, 2008 Why is InsanelyMac so slow these days? It takes forever to open pages (Waiting for a.fubra.com or Waiting for forum.insanelymac.com) and its not my net connection Link to comment Share on other sites More sharing options...
apowerr Posted January 1, 2008 Share Posted January 1, 2008 Same issue here. And the frequent IPS driver errors have been driving me insane. Link to comment Share on other sites More sharing options...
Griffin Lougerone Posted January 1, 2008 Share Posted January 1, 2008 sadly its true, and i think search function doesnt work well either Link to comment Share on other sites More sharing options...
~pcwiz Posted January 1, 2008 Author Share Posted January 1, 2008 Either the page takes forever to load or I get a "IPS Driver Error" or some server load error. Time to upgrade the servers... Search has never worked well on InsanelyMac Link to comment Share on other sites More sharing options...
Griffin Lougerone Posted January 1, 2008 Share Posted January 1, 2008 plus it is slow on responses too, many people view the topics but a rare case if anyone replies, so its slow in helping too Link to comment Share on other sites More sharing options...
~pcwiz Posted January 1, 2008 Author Share Posted January 1, 2008 I know administration is trying as hard as they can, but all these problems (other than search ) are caused by too much server load because so many users are using the forum. Even if they don't upgrade now, they WILL have to upgrade the servers in the near future to keep this place alive. Link to comment Share on other sites More sharing options...
skyhighmac Posted January 1, 2008 Share Posted January 1, 2008 Everyone click on an ad to pay for the new servers! Come on guys, let's go!!!! Link to comment Share on other sites More sharing options...
SticMAC™ Posted January 1, 2008 Share Posted January 1, 2008 Maybe if people started READING the Posts instead of POSTING new one's, (RTFM's) then the servers will be able to handle it for a few good years more! AND O,yes if people stop posting {censored} topics like "how's my toenails looking?, or "lets see who can make coffee the fastest?", or "just how id you get you colour blue to look like green?" or "mine's MORE official and MORE expensive to build than yours!!" OR ask DUMB-ASS questions like "Why is InsanelyMac so slow these days?" ---*jeeeshzzz*--- SticMAN ps "DUMB-ASS" courtesy of That seventies Show Link to comment Share on other sites More sharing options...
Numberzz Posted January 1, 2008 Share Posted January 1, 2008 My T.V. looks a bit lopsided... Link to comment Share on other sites More sharing options...
ComplexMind Posted January 1, 2008 Share Posted January 1, 2008 Hi Everyone Mark from Fubra here, I have been monitoring the site all over the Xmas break. We have a monitoring system which checks the site every 30 seconds and reports to me and the other server admins when a problem is found. This system traps slow page loads, page errors and database errors as well as server health issues such as cpu and memory loads. With any site serving tens or hundreds thousands of page requests per day it is normal to receive the occasional alert - this does not necessarily mean there is a major problem. The servers hosting this site have indeed been very busy over the last few days but I have received very few errors: [05:04:04 AM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: ON [05:04:34 AM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: OFF [06:34:02 AM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: ON [06:34:32 AM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: OFF [11:34:11 AM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: ON [11:34:41 AM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: OFF [02:34:13 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: ON [02:34:43 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: OFF [03:34:15 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: ON [03:35:15 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: OFF [04:04:15 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: ON [04:05:15 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: OFF [04:34:17 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: ON [04:35:17 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: OFF [05:04:02 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: ON [05:05:03 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: OFF [05:34:33 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: ON [05:35:33 PM] zabbix: forum.insanelymac.com response code not 200 from XXX.bunker.fubra.net: OFF These errors suggest to me that something is happening around 4 mins past each half hour, but there are no scheduled tasks running on the server at these times so what is causing these issues remains, at present, a mystery. What I will say though, is that every time I have been at my computer when the alert is triggered, I have browsed the site to try and replicate the error, so far without success. Every time, the page loads quickly and completely. No IPS errors or anything.... So yes, it is a bit of a mystery. No, the servers aren't overloaded. No, the database is not overloaded (runs around 20% cpu during heavy periods). And no the webserver is not overloaded either... In fact this page loaded for me in less than a second over a crappy NTL connection. I have posted in all this detail just so that you know somebody is watching and that we do care about keeping the site quick and error free. I will keep monitoring the situation and do my best to pin down the cause. I have had a few ideas about getting more fine-grained information out of the webserver, but with the page requests hitting this webserver it's most definitely it's an "electron in a macstack"... In the time being, refreshing the page would seem to be the only way around it, so refresh away. The servers can handle it. Happy New Year to everyone on the board, I'll update as soon as I get to the bottom of the problem! Link to comment Share on other sites More sharing options...
skyhighmac Posted January 1, 2008 Share Posted January 1, 2008 Thanks Mark. It seems there are "periods" of when you get driver errors. It seems like one day you'll get ten and you wont get any for a while. Link to comment Share on other sites More sharing options...
Numberzz Posted January 1, 2008 Share Posted January 1, 2008 I JUST FIGURED IT OUT!!!!!!!! Maybe... Don't worry, not related to anything anyone needs to be worried about :mellow: Link to comment Share on other sites More sharing options...
ComplexMind Posted January 1, 2008 Share Posted January 1, 2008 I found it... :mellow: It's the Sphinx indexer program... Sphinx is a full-text indexing system which crawls the database every half hour and builds a text index of key database tables so it can return user searches more quickly without hammering the database. Now I've found it I can do something about it. Watch this space.... M. Link to comment Share on other sites More sharing options...
skyhighmac Posted January 1, 2008 Share Posted January 1, 2008 Thank you Mark! Link to comment Share on other sites More sharing options...
Suhail Posted January 1, 2008 Share Posted January 1, 2008 Just wanted to add my experience, The site does load up quickly for me (Also using a crappy NTHell (NTL) connection). But the IPS Driver Error is a problem which also exists for meh. Link to comment Share on other sites More sharing options...
A Nonny Moose Posted January 1, 2008 Share Posted January 1, 2008 I tend to get nothing but "IPS Driver Error" whenever I come here lately. Link to comment Share on other sites More sharing options...
mifki Posted January 2, 2008 Share Posted January 2, 2008 Mark, the Database indexer shouldnt make such a large difference, any enterprise class server should be able to handle multiple indexes while serving dynamic pages. I was doing some tests and its always the ad subdomain which has the highest response time, maybe some optimisations might be needed on that server, they are in the same datacenter, so it cant be a location issue. If you think its the db indexer, set the cron job to work every 1 hour and see if the performance improves P.S: I suggest starting a special poll which can be voted by a user everyday/multiple times a day asking how well the site has been performing, that way it can help you keep track of high load times, and schedule any cron jobs for the low peaks P.P.S: wasting tons of time pcwiz? i thought that was you and your posting useless junk 300 times a day. note to the mod who keeps editing my posts, grow up. Editing things out just because you dont like what it says is completely unprofessional and an abuse of the power you have been given. Link to comment Share on other sites More sharing options...
CLiDE FTW!!1 Posted January 2, 2008 Share Posted January 2, 2008 Search should just be removed all together from this board, it basically ignores the word you're searching for and spits out random results. Link to comment Share on other sites More sharing options...
ComplexMind Posted January 2, 2008 Share Posted January 2, 2008 OK, just thought I'd give an update on the ips driver issue. I have looked into it in detail today and there is no doubt that the situation can be resolved with a few basic steps The periodical indexing process which feeds the full-text search engine we use on this site is most certainly the trigger of the problem since it is completely locking down the posts and comments tables when it runs. The indexer is not the core of the problem though, there are a few factors going on as far as I can see. First, the mysql server had a ridiculously low max connections limit. The sheer number of requests hitting the server is causing the server to refuse connections when the tables become locked for indexing and the connection back-log builds up. I have increased the maximum connection limit to a more realistic value and if necessary will increase it again - who knows it could even improve the overall performance of the site during busy periods but it should certainly smooth over the indexing lock a little better. Second, I have discovered that these tables being locked for indexing are currently using the myisam engine which locks at a table level rather than at row level. This means that when the index runs it locks the whole table instead of just the rows it is indexing on that run. I am going to look into converting the table to innodb but it's not something I'll do in a rush. I'm going to do a little more homework and speak to a few people and hopefully convert in the next few days. Of course, this will mean some downtime, hopefully not too much Thanks to everyone who has offered help on the matter (Suhail and Kiko)... Again, watch this space. M. Link to comment Share on other sites More sharing options...
ComplexMind Posted January 2, 2008 Share Posted January 2, 2008 Oh, and I also fixed the server time problem permanently. The underlying host box is now linked to our time servers. Some of the site totals may look funny or skewed for the next few hours... EDIT: Oops looks like someone had put in a manual board time offset to counteract the incorrect system time manually, by 57 seconds... How random is that. I have this mental image of somebody logging in every day, working out how 'wrong' the server was and updating the board time offset accordingly- brilliant! M. Link to comment Share on other sites More sharing options...
Numberzz Posted January 2, 2008 Share Posted January 2, 2008 It's off by 1 hour 3 minutes. What do you mean by 'site totals?' EDIT: It's right again. Link to comment Share on other sites More sharing options...
ComplexMind Posted January 2, 2008 Share Posted January 2, 2008 It's off by 1 hour 3 minutes. What do you mean by 'site totals?'EDIT: It's right again. I meant things like active viewer counts... Link to comment Share on other sites More sharing options...
mifki Posted January 3, 2008 Share Posted January 3, 2008 Its quicker now, I also recommend splitting the db into 2 parts considering it is rahter large and contains heaps of threads which havent been replied to in over 6 months and are irrelevant (10.4.1 etc etc), they can be placed into a static archive to reduce cpu load further Link to comment Share on other sites More sharing options...
Mebster Posted January 3, 2008 Share Posted January 3, 2008 Its quicker now, I also recommend splitting the db into 2 parts considering it is rahter large and contains heaps of threads which havent been replied to in over 6 months and are irrelevant (10.4.1 etc etc), they can be placed into a static archive to reduce cpu load further Kiko, so there is a nice and helpful side to you after all Good to see you behaving mate. Link to comment Share on other sites More sharing options...
ComplexMind Posted January 3, 2008 Share Posted January 3, 2008 Its quicker now, I also recommend splitting the db into 2 parts considering it is rahter large and contains heaps of threads which havent been replied to in over 6 months and are irrelevant (10.4.1 etc etc), they can be placed into a static archive to reduce cpu load further Yeah, fair point. But you'll have to trust me on this one - the database has a dedicated box with dual 3.02GHz Xeons and 4GB of ram. It runs around 100-200qps at between 10-30% cpu utilisation during busy periods. The database isn't that big really, it's well indexed and mysql just rocks For such a busy site, mysql is doing a great job - despite the perception caused by the admittedly nasty IPS errors. But, with the increased max connections in mysql the indexer run doesn't seem to have caused any ips errors in the last few hours - if anyone spots one then please let me know! Finally, I re-enabled the webserver option for keepalives which was disabled for performance reasons months ago when the site took a very severe battering on a popular inbound link from somewhere. This should already be making a huge difference to page load times for our US members "crossing the pond" to get their InsanelyMac fix. M. Link to comment Share on other sites More sharing options...
Recommended Posts