Turning WP blogs into HTML

[TabbyGirl]

Is it a good idea, mine are being slowed by bots and it's killing me on bandwidth usage, what do you think.

[Wm2man]

You can try Simply static

[tacnet]

Its very easy to setup a cron job to read your high traffic pages and store them as static html, say every minute.

Then its simply a case of using .htaccess to redirect links to those static html files

Its a standard scaling up technique that we've been using for years

[kdz]

If you have a few pages you can do html(static), it will load also faster.

[Affman]

WP is good keep it updated and limit plugins it works fine.

[jag]

WP works why mess around with it

[tacnet]

WP works why mess around with it

Its susceptible to bot attacks. I don't use WP but I assume the pages are all dynamic ?? If so, then a bot hammering those pages can really slow down your server or crash it altogether. Just like the problems Butcher has had on this forum

I recommend getting an account with https://gtmetrix.com/. They can measure your page load speed every hour and alert you when the bots start to slow things down. They also give you a lot of tips on how to maximize your page load speed

[Matty2]

It it ain't broken don't fix it. Wordpress is good let it be.

[tacnet]

It it ain't broken don't fix it. Wordpress is good let it be.

It's not about WP. Wordpress is fine but, like any other script (including this forum), the pages are dynamic. That means every time a page loads, there's a whole load of MySql commands that have to be executed in the background. Now if you have a lot of traffic (or your getting hammered by bots thousands of times a second), then all those MySql commands will bring the website to a grinding halt so it's a good idea to make your high traffic pages into static html

For example, our index page probably gets around 200k hits a day and if that was a dynamic page, it would slow the whole site down and loose us business. So instead we make it static html and update it (along with a whole bunch of other pages) every minute or so.

[nudelinks]

How does that work when you want to add a new post?

[tacnet]

How does that work when you want to add a new post?

You can just carry on using WP as normal. Make your posts and the cron job will automatically update the html pages every minute (or however often you set it to run)

Below is a copy of a post I did in the other thread regarding bots. It relates to pages on this forum but it can be tweaked to whatever pages you need

ok this is quick and dirty. Cron job will look something like this :-

Code: Select all

<?php 
//   First write the index to index.html
$html = file_get_contents('https://www.sponsorchat.com/index.php');
$filename = 'index.html';
$handle = fopen($filename,"w");
fwrite($handle,$html);
fclose($handle);

//   Now write topic list to topics.html
$html = file_get_contents('https://www.sponsorchat.com/viewforum.php?f=74');
$filename = 'topics.html';
$handle = fopen($filename,"w");
fwrite($handle,$html);
fclose($handle);
?>

Then, if your not familiar with editing your crontab file then I would get your hosts to add this line to it

Code: Select all

* * * * * php /full/path/to/your/php/file/mycronfile.php

That will run your cron file once every minute. The * * * * * means run the cron every minute. Or I could use */10 * * * * to run it every 10 minutes

Hope that helps

[gotonudecom]

WP works why mess around with it

Its susceptible to bot attacks. I don't use WP but I assume the pages are all dynamic ?? If so, then a bot hammering those pages can really slow down your server or crash it altogether. Just like the problems Butcher has had on this forum

I recommend getting an account with https://gtmetrix.com/. They can measure your page load speed every hour and alert you when the bots start to slow things down. They also give you a lot of tips on how to maximize your page load speed

agree,
Except caching.
Bikker upgraded our caching last week when several sites were getting pounded by Singapore bots.
Then he put some type of IP limiter on the server side.

You can not expect your CMS to deal with all this, especially bots.
that is NOT the job of the CMS/Wordpress. The server is responsible for dealing with attacks, not the CMS.

Every site is susceptible to bot attacks!

I do not think this site is under attack, I believe that some scrapper is just having its way with the site, due to the fact safe guards were not in place, server side.
At least that is what we have determined from the Singapore traffic that is hitting several of my sites and my client sites.

The real question is, after they are done scrapping do they stop? after they stop can any benefit of their scrapping be realized. for instance does traffic from some AI increase or some tiny or new search engine.
The price is a little discomfort when your site is slow. Only until you teach the SERVER to deal with the bots.

Tac please comment and share with me your thoughts. I really like to be correct. The only way to be correct is when someone shares information to update your knowledge.

[tacnet]

WP works why mess around with it

Its susceptible to bot attacks. I don't use WP but I assume the pages are all dynamic ?? If so, then a bot hammering those pages can really slow down your server or crash it altogether. Just like the problems Butcher has had on this forum

I recommend getting an account with https://gtmetrix.com/. They can measure your page load speed every hour and alert you when the bots start to slow things down. They also give you a lot of tips on how to maximize your page load speed

agree,
Except caching.
Bikker upgraded our caching last week when several sites were getting pounded by Singapore bots.
Then he put some type of IP limiter on the server side.

You can not expect your CMS to deal with all this, especially bots.
that is NOT the job of the CMS/Wordpress. The server is responsible for dealing with attacks, not the CMS.

Every site is susceptible to bot attacks!

I do not think this site is under attack, I believe that some scrapper is just having its way with the site, due to the fact safe guards were not in place, server side.
At least that is what we have determined from the Singapore traffic that is hitting several of my sites and my client sites.

The real question is, after they are done scrapping do they stop? after they stop can any benefit of their scrapping be realized. for instance does traffic from some AI increase or some tiny or new search engine.
The price is a little discomfort when your site is slow. Only until you teach the SERVER to deal with the bots.

Tac please comment and share with me your thoughts. I really like to be correct. The only way to be correct is when someone shares information to update your knowledge.

Your absolutely right. Your CMS/Wordpress won't help you when it comes to bots. Anything that is dynamic (ie not a flat HTML page) can really load up your servers once the bots start hitting it. Cloudfare is one solution BUT bear in mind that it will block ALL bots, including google, bing etc

Adjusting your cache policies might help a little but bots tend to ignore cache and just keep hammering away at the same page. And Ive seen a lot of them spoofing the IP address so each hit comes from a different IP making it impossible to block

The only solution we have found is using a cron job to convert dynamic pages into flat HTML pages. We've even gone so far as to create HTML snippets that can be easily included into a page rather than querying the database. For example https://www.tacamateurs.com/sites/sugarbabe/ . The "Trending Updates", "Latest Updates", "Popular TAC Websites", "Recently Updated TAC Websites" and the "Our Reviews" sections are all HTML snippets that we update every 5 minutes. It means that page puts very little strain on the Sql server no matter how often the bots hammer it

[tacnet]

One other thing worth mentioning, flat HTML pages respond much quicker than dynamic pages. That means your page load speed goes up which google likes a lot

[gotonudecom]

Its susceptible to bot attacks. I don't use WP but I assume the pages are all dynamic ?? If so, then a bot hammering those pages can really slow down your server or crash it altogether. Just like the problems Butcher has had on this forum

I recommend getting an account with https://gtmetrix.com/. They can measure your page load speed every hour and alert you when the bots start to slow things down. They also give you a lot of tips on how to maximize your page load speed

agree,
Except caching.
Bikker upgraded our caching last week when several sites were getting pounded by Singapore bots.
Then he put some type of IP limiter on the server side.

You can not expect your CMS to deal with all this, especially bots.
that is NOT the job of the CMS/Wordpress. The server is responsible for dealing with attacks, not the CMS.

Every site is susceptible to bot attacks!

I do not think this site is under attack, I believe that some scrapper is just having its way with the site, due to the fact safe guards were not in place, server side.
At least that is what we have determined from the Singapore traffic that is hitting several of my sites and my client sites.

The real question is, after they are done scrapping do they stop? after they stop can any benefit of their scrapping be realized. for instance does traffic from some AI increase or some tiny or new search engine.
The price is a little discomfort when your site is slow. Only until you teach the SERVER to deal with the bots.

Tac please comment and share with me your thoughts. I really like to be correct. The only way to be correct is when someone shares information to update your knowledge.

Your absolutely right. Your CMS/Wordpress won't help you when it comes to bots. Anything that is dynamic (ie not a flat HTML page) can really load up your servers once the bots start hitting it. Cloudfare is one solution BUT bear in mind that it will block ALL bots, including google, bing etc

Adjusting your cache policies might help a little but bots tend to ignore cache and just keep hammering away at the same page. And Ive seen a lot of them spoofing the IP address so each hit comes from a different IP making it impossible to block

The only solution we have found is using a cron job to convert dynamic pages into flat HTML pages. We've even gone so far as to create HTML snippets that can be easily included into a page rather than querying the database. For example https://www.tacamateurs.com/sites/sugarbabe/ . The "Trending Updates", "Latest Updates", "Popular TAC Websites", "Recently Updated TAC Websites" and the "Our Reviews" sections are all HTML snippets that we update every 5 minutes. It means that page puts very little strain on the Sql server no matter how often the bots hammer it

That is a really great way to handle this. thanks for sharing.