Association Forum - Association Forum - Crash and Data Loss 6-Oct-2020

Assn Members Area · Join

Home.

Association Forum

Ships Store

Cruising

Racing

Technical Tips

Photo Gallery - New

Boat Search - Updated

Manuals & Brochures

Swap Meet

Links

Officer Flag Line & Staff

Bylaws & Constitution

Members Area

Join the Association

Notice:
The advice given on this site is based upon individual or quoted experience, yours may differ.
The Officers, Staff and members of this site only provide information based upon the concept that anyone utilizing this information does so at their own risk and holds harmless all contributors to this site.

Association Forum

Username:	Password:
Save Password
Forgot your Password?

All Forums

Association Business

International Association Business

Association Forum - Crash and Data Loss 6-Oct-2020

New Topic

Topic Locked

Printer Friendly

Author

Topic

Commodore
Russ Johnson

Member Avatar

USA
695 Posts

Initially Posted - 10/08/2020 : 22:44:37

To All,

Tuesday, 6-Oct-2020 the forum crashed.
While the website itself was available, the forum application could not connect to the database.
Meaning the forum was down and users received an error messages when trying to access the forum.

After working with the website hosting company, there has been an unrecoverable data loss.
All forum data from 17-Jun-2020 to 6-Oct-2020 has been permanently lost.

Here are the details.
On 17-Jun-2020, the website hosting company migrated our Windows domain (catalina-capri-25s.org) to a new server.
The new server has newer software and an improved admin tool called "Plesk".
Along with the domain migration, the website hosting company also moved the MySQL database to a new server.
In the past, these upgrades are usually seamless.
In fact, the notice from the website hosting company said "... there is nothing for you to do".

After the migration, we confirmed the website and forum were working properly.
The forum continued to operate until 6-Oct-2020.

What was not known was on 17-Jun-2020, the MySQL database was copied to a new server not moved.
This means there were two copies of the database.
The forum application on the new server was still using the database on the old server.

The Plesk admin tool showed the database was running and the admin user could view the tables and data.
Also, the Plesk admin tool would backup the database, but it was connected to the copy from 17-Jun-2020.
There are multiple backups of the same database copy from 17-Jun-2020.

On 6-Oct-2020, the old database server was shutdown and unplugged.
That is why the forum errors suddenly appeared.
The website hosting company does not have any database backups from the old servers.

We have recovered the forum using the MySQL database from the 17-Jun-2020.
All forum data from 17-Jun-2020 to 6-Oct-2020 has been permanently lost.

We are truly sorry for this situation.
Russ Johnson
Commodore

ps. The Members Area database was not impacted.
If you posted photos to the Members Area gallery they are still available.

Russ Johnson, Commodore 2009-2024 (commodore at catalina-capri-25s.org)

Voyager
Master Marine Consultant

Members Avatar

USA
5313 Posts

Response Posted - 10/09/2020 : 06:26:58

Hello Russ, thank you for providing the account of what took place. It�s a shame that the forum has lost the great info that was presented by the members.
The good news is that we�re all still here, the Forum is up and running and we can repost the lost topics if we wish..
As I recall there were topics ranging from replacing the turning ball for a swing keel boat, refurbishing an older boat that had been neglected, replacing the head and holding tank, rewiring the deck mast connector, and purchasing new cushions. And of course, Ray posted about the big storm on Lake Lanier, and later a photo of an ironically peaceful moment one evening on the lake.
There were dozens more threads of course. We can�t go back so let�s just move forward.
Now we�re heading into the winter season up north, so there�s plenty of tech talk to cover for maintenance topics.
And in Texas and Florida, the super hot, muggy weather is finally breaking, so we should get some great sailing stories.
It�s a great resource we have here, let�s make the best of it.

Bruce Ross
Passage ~ SR-FK ~ C25 #5032

Port Captain � Milford, CT

Commodore
Russ Johnson

Members Avatar

USA
695 Posts

Response Posted - 10/11/2020 : 13:43:51

Good idea Bruce.
Yes, photos posted to the Members Area are still there.
Photos posted to 3rd-party sites are still there.
We don't have the posts or the links to the photos.

It would be nice if users could post their stories again and link the their photos.

Russ Johnson, Commodore 2009-2024 (commodore at catalina-capri-25s.org)

Commodore
Russ Johnson

Members Avatar

USA
695 Posts

Response Posted - 10/11/2020 : 14:57:15

Here's a copy of my reply to a post on the C25 Forum.
http://www.catalina-capri-25s.org/forum/topic.asp?TOPIC_ID=31933

Hey guys,

These are all good questions.
Here are some clarifications from my other post.
You can post comments there.
Association Forum - Crash and Data Loss Oct-6
http://www.catalina-capri-25s.org/forum/topic.asp?TOPIC_ID=31937

We have weekly backups of the forum database.
The problem is, every week the website company was backing up the same file from Jun-17.
There are several copies of the Jun-17 database file.

The notice we received:
The website was moving to a new server.
There is a new website admin tool call "Plesk".
There is nothing needed on your part to complete the migration.

What was not in the notice.
Moving the website also means copying the database to a new server.
They didn't communicate there was two copies of the database running at the same time.
They didn't communicate we needed to manually disconnect from the old database server and connect to the new database server.
They didn't communicate they were shutting down the old database server.

It is standard practice to keep websites and databases on separate servers.
That was the configuration before and the configuration now.
The notice only mentions the website and there is no mention of the database.

Next, it is common to connect to a database by either the server name or IP address.
We are using the server name. In the Plesk tool, it only displays the IP address.
When "moving" a database to a new server, IT "usually" updates the server name to point to the new IP address.
This is a database alias, which they didn't update or communicate they were not going to update.

I've been through many IT projects in my career, but this company didn't follow standard practices.
Examples:
When two copies of the same database are running, check if anyone is connected to the old database.
Update the database alias to the new IP address.
Before shutting down a database, make it unavailable. Change the connection string, change the database name, change the password, anyone one of those will do.
Before unplugging a server, stop the services. In this case stop the MySQL database service.
Once you follow any of those steps, you let the machine run for a few days.
In our case, 3-months.
The website company didn't do any of those things.
They copied the database to a new server in June, they didn't check anything, they shutdown and unplugged the machine on the same day.

Using the Plesk admin tool:
You could see the database backups, it was not clear this was the database copy from Jun-17.
You could see the database was running, it was not clear this was the database copy from Jun-17.
You could run queries and see forum data, it was not clear this was data from Jun-17.

In hindsight, had we spent more time using the Plesk admin tool, we could have run queries to see the most recent forum post or see the most recent forum user.
Then determine new data was not being added to the new database (had we known there was a new database).
Since the forum was running well, this about of troubleshooting was not considered.

Thank you for your comments.
I'm going through the 5-stages of grief.
I may be upset, but not at you or the forum users.
Thank you for your understanding.

Russ Johnson, Commodore 2009-2024 (commodore at catalina-capri-25s.org)

Topic

New Topic

Topic Locked

Printer Friendly

Jump To:

Association Forum

Notice: The advice given on this site is based upon individual or quoted experience, yours may differ.
The Officers, Staff and members of this site only provide information based upon the concept that anyone utilizing this information does so at their own risk and holds harmless all contributors to this site.