The advice given on this site is based upon individual or quoted experience, yours may differ.
The Officers, Staff and members of this site only provide information based upon the concept that anyone utilizing this information does so at their own risk and holds harmless all contributors to this site.
Note: You must be registered in order to post a reply. To register, click here. Registration is FREE!
T O P I C R E V I E W
Posted - 10/08/2020 : 22:44:37 To All,
Tuesday, Oct-6 the forum crashed. While the website itself was available, the forum application could not connect to the database. Meaning the forum was down and users received an error messages when trying to access the forum.
After working with the website hosting company, there has been an unrecoverable data loss. All forum data from Jun-17 to Oct-6 has been permanently lost.
Here are the details. On Jun-17, the website hosting company migrated our Windows domain (catalina-capri-25s.org) to a new server. The new server has newer software and an improved admin tool called "Plesk". Along with the domain migration, the website hosting company also moved the MySQL database to a new server. In the past, these upgrades are usually seamless. In fact, the notice from the website hosting company said "... there is nothing for you to do".
After the migration, we confirmed the website and forum were working properly. The forum continued to operate until Oct-6.
What was not known was on Jun-17, the MySQL database was copied to a new server not moved. This means there were two copies of the database. The forum application on the new server was still using the database on the old server.
The Plesk admin tool showed the database was running and the admin user could view the tables and data. Also, the Plesk admin tool would backup the database, but it was connected to the copy from Jun-17. There are multiple backups of the same database copy from Jun-17.
On Oct-6, the old database server was shutdown and unplugged. That is why the forum errors suddenly appeared. The website hosting company does not have any database backups from the old servers.
We have recovered the forum using the MySQL database from the Jun-17. All forum data from Jun-17 to Oct-6 has been permanently lost.
We are truly sorry for this situation. Russ Johnson Commodore
ps. The Members Area database was not impacted. If you posted photos to the Members Area gallery they are still available.
We have weekly backups of the forum database. The problem is, every week the website company was backing up the same file from Jun-17. There are several copies of the Jun-17 database file.
The notice we received: The website was moving to a new server. There is a new website admin tool call "Plesk". There is nothing needed on your part to complete the migration.
What was not in the notice. Moving the website also means copying the database to a new server. They didn't communicate there was two copies of the database running at the same time. They didn't communicate we needed to manually disconnect from the old database server and connect to the new database server. They didn't communicate they were shutting down the old database server.
It is standard practice to keep websites and databases on separate servers. That was the configuration before and the configuration now. The notice only mentions the website and there is no mention of the database.
Next, it is common to connect to a database by either the server name or IP address. We are using the server name. In the Plesk tool, it only displays the IP address. When "moving" a database to a new server, IT "usually" updates the server name to point to the new IP address. This is a database alias, which they didn't update or communicate they were not going to update.
I've been through many IT projects in my career, but this company didn't follow standard practices. Examples: When two copies of the same database are running, check if anyone is connected to the old database. Update the database alias to the new IP address. Before shutting down a database, make it unavailable. Change the connection string, change the database name, change the password, anyone one of those will do. Before unplugging a server, stop the services. In this case stop the MySQL database service. Once you follow any of those steps, you let the machine run for a few days. In our case, 3-months. The website company didn't do any of those things. They copied the database to a new server in June, they didn't check anything, they shutdown and unplugged the machine on the same day.
Using the Plesk admin tool: You could see the database backups, it was not clear this was the database copy from Jun-17. You could see the database was running, it was not clear this was the database copy from Jun-17. You could run queries and see forum data, it was not clear this was data from Jun-17.
In hindsight, had we spent more time using the Plesk admin tool, we could have run queries to see the most recent forum post or see the most recent forum user. Then determine new data was not being added to the new database (had we known there was a new database). Since the forum was running well, this about of troubleshooting was not considered.
Thank you for your comments. I'm going through the 5-stages of grief. I may be upset, but not at you or the forum users. Thank you for your understanding.
Posted - 10/11/2020 : 13:43:51 Good idea Bruce. Yes, photos posted to the Members Area are still there. Photos posted to 3rd-party sites are still there. We don't have the posts or the links to the photos.
It would be nice if users could post their stories again and link the their photos.
Posted - 10/09/2020 : 06:26:58 Hello Russ, thank you for providing the account of what took place. Itís a shame that the forum has lost the great info that was presented by the members. The good news is that weíre all still here, the Forum is up and running and we can repost the lost topics if we wish.. As I recall there were topics ranging from replacing the turning ball for a swing keel boat, refurbishing an older boat that had been neglected, replacing the head and holding tank, rewiring the deck mast connector, and purchasing new cushions. And of course, Ray posted about the big storm on Lake Lanier, and later a photo of an ironically peaceful moment one evening on the lake. There were dozens more threads of course. We canít go back so letís just move forward. Now weíre heading into the winter season up north, so thereís plenty of tech talk to cover for maintenance topics. And in Texas and Florida, the super hot, muggy weather is finally breaking, so we should get some great sailing stories. Itís a great resource we have here, letís make the best of it.
Notice: The advice given on this site is based upon individual or quoted experience, yours may differ. The Officers, Staff and members of this site only provide information based upon the concept that anyone utilizing this information does so at their own risk and holds harmless all contributors to this site.