[p4] How do i find corrupt depot files

[Anders Kjærgaard Hansen]

Hi

I have just found a corrupt file in our depot.

The incident has happened just as we have switched server, and had some trouble with a snapshot file from a VM image.

I would very much be able to query our p4 server to go through all files tonight, and report all files which have problems with its files.

Can anyone help me with an idea on how to script this, or use an existing p4 command to scan our depot, and report all files that are corrupt?

I hope for the best that it is only this one file.

/Anders

[Stephen Vance]

Look at 'p4 verify -q //...'. Ckeck the Admin Guide for more details.

Stephen Vance

Post by Anders Kjærgaard Hansen
Hi
I have just found a corrupt file in our depot.
The incident has happened just as we have switched server, and had
some trouble with a snapshot file from a VM image.
I would very much be able to query our p4 server to go through all
files tonight, and report all files which have problems with its
files.
Can anyone help me with an idea on how to script this, or use an
existing p4 command to scan our depot, and report all files that are
corrupt?
I hope for the best that it is only this one file.
/Anders
_______________________________________________
http://maillist.perforce.com/mailman/listinfo/perforce-user

[Michael Delaney]

Switched over from what to what?

You can use the 'p4 verify' (see 'p4 help verify') command to check that
the files hashes are as expected.

Post by Anders Kjærgaard Hansen
Hi
I have just found a corrupt file in our depot.
The incident has happened just as we have switched server, and had some trouble with a snapshot file from a VM image.
I would very much be able to query our p4 server to go through all files tonight, and report all files which have problems with its files.
Can anyone help me with an idea on how to script this, or use an existing p4 command to scan our depot, and report all files that are corrupt?
I hope for the best that it is only this one file.
/Anders
_______________________________________________
http://maillist.perforce.com/mailman/listinfo/perforce-user

[Anders Kjærgaard Hansen]

I actually tried verify on the corrupt file, and didn't get the expected result.

Or maybe I was typing something wrong.

I will do a full verify soon.

The story short here I guess, I that we had our server on a 2008 R2, running in vmware. We then experience some trouble with R2, installed a new 2008, and moved the virtual disk to the new server.

In doing this some snapshot broke, and got trashed somehow.

So the things I see know on our server seems to bit a corrupt FAT table or the like.

A submitted files content has just ended up in our license file.

So I have shut down the server, added a new disk, and are now in the process of copying all data to the new disk, and will then start to recover.

I expect P4 verify will do the trick in aiding me when it comes to all the depot files.

I'm not entirely sure here, but if I just run 'p4 verify //...#1,#1'. Will I then be able to find all files where the actual file is corrupted, e.g. not valid syntax. The errors I will experience will most likely be syntax/parse errors, and those will almost always exist in #1. Am I right?

Thanks for your quick replies.

/Anders



-----Original Message-----
From: Michael Delaney [mailto:***@upromise.com]
Sent: Thursday, October 01, 2009 3:10 PM
To: Anders Kjærgaard Hansen
Cc: perforce-***@perforce.com
Subject: Re: [p4] How do i find corrupt depot files

Switched over from what to what?

You can use the 'p4 verify' (see 'p4 help verify') command to check that
the files hashes are as expected.

Post by Anders Kjærgaard Hansen
Hi
I have just found a corrupt file in our depot.
The incident has happened just as we have switched server, and had some trouble with a snapshot file from a VM image.
I would very much be able to query our p4 server to go through all files tonight, and report all files which have problems with its files.
Can anyone help me with an idea on how to script this, or use an existing p4 command to scan our depot, and report all files that are corrupt?
I hope for the best that it is only this one file.
/Anders
_______________________________________________
http://maillist.perforce.com/mailman/listinfo/perforce-user

[s***@vance.com]

So first of all, you switched servers with no change in operating system,
OS version, or OS bit-ness, right?

This sounds like an ideal time to use your backups. You have backups of
your Perforce data, right? If you don't, make it a high priority
immediately after you recover from this situation. Put it on a regular
schedule that reflects your maximum acceptable data loss and regularly
verify that your backup procedure is working correctly. See the Admin Guide
for best practices on backup.

That aside, you wouldn't need to use "#1,#1", just "#1". That form of
verify only verifies the first revision, which only serves as a reasonable
fast shortcut to verify the whole history for text files, not binary files.
You don't say whether the file in question is binary or text, so make sure
it's a text file before taking the shortcut.

Verify doesn't check the "syntax" of the archive file or of the code in the
revision directly. Rather, it constructs the revision being verified and
compares the MD5 checksum of that revision to the checksum that was stored
in the metadata when it was submitted or, in some cases, when the checksum
was computed separate from submission.

So the submitted file that ended up in your license file, was it submitted
before or after the server switch? If after, the problem may be only with
the FAT table around your license file and not the archive file. However,
in that circumstance, a copy may not give you good data even if it makes
sure you have a good file system integrity.

Advice: Don't use FAT*. It's gotten better, but it's not a very robust file
system.

Original Message:
-----------------
From: Anders Kjærgaard Hansen ***@etiglobal.net
Date: Thu, 1 Oct 2009 15:36:48 +0200
To: ***@upromise.com, perforce-***@perforce.com
Subject: Re: [p4] How do i find corrupt depot files


I actually tried verify on the corrupt file, and didn't get the expected
result.

Or maybe I was typing something wrong.

I will do a full verify soon.

The story short here I guess, I that we had our server on a 2008 R2,
running in vmware. We then experience some trouble with R2, installed a new
2008, and moved the virtual disk to the new server.

In doing this some snapshot broke, and got trashed somehow.

So the things I see know on our server seems to bit a corrupt FAT table or
the like.

A submitted files content has just ended up in our license file.

So I have shut down the server, added a new disk, and are now in the
process of copying all data to the new disk, and will then start to recover.

I expect P4 verify will do the trick in aiding me when it comes to all the
depot files.

I'm not entirely sure here, but if I just run 'p4 verify //...#1,#1'. Will
I then be able to find all files where the actual file is corrupted, e.g.
not valid syntax. The errors I will experience will most likely be
syntax/parse errors, and those will almost always exist in #1. Am I right?

Thanks for your quick replies.

/Anders



-----Original Message-----
From: Michael Delaney [mailto:***@upromise.com]
Sent: Thursday, October 01, 2009 3:10 PM
To: Anders Kjærgaard Hansen
Cc: perforce-***@perforce.com
Subject: Re: [p4] How do i find corrupt depot files

Switched over from what to what?

You can use the 'p4 verify' (see 'p4 help verify') command to check that
the files hashes are as expected.

Post by Anders Kjærgaard Hansen
Hi
I have just found a corrupt file in our depot.
The incident has happened just as we have switched server, and had some

trouble with a snapshot file from a VM image.

Post by Anders Kjærgaard Hansen
I would very much be able to query our p4 server to go through all files

tonight, and report all files which have problems with its files.

Post by Anders Kjærgaard Hansen
Can anyone help me with an idea on how to script this, or use an existing

p4 command to scan our depot, and report all files that are corrupt?

Post by Anders Kjærgaard Hansen
I hope for the best that it is only this one file.
/Anders
_______________________________________________
http://maillist.perforce.com/mailman/listinfo/perforce-user

_______________________________________________
perforce-user mailing list - perforce-***@perforce.com
http://maillist.perforce.com/mailman/listinfo/perforce-user


--------------------------------------------------------------------
mail2web LIVE – Free email based on Microsoft® Exchange technology -
http://link.mail2web.com/LIVE

[Tim McDaniel]

Post by Anders Kjærgaard Hansen
I would very much be able to query our p4 server to go through all
files tonight, and report all files which have problems with its
files.

Someone mentioned "p4 verify -q //...".

I didn't like that so much because it takes 2 hours or so on my
workplace's repository, and I believe that it locks against submits
the entire time.

I did a script to somewhat ameliorate that. I'd post it except
- my job claims ownership of all code done for the job
- it's straightforward to do if you know how to script in
UNIX-like systems (or Cygwin).

The outline:
- renice $$ to a lower priority
- p4 files | sed -e 's/#.*//' > files
- use the split command to split files into chunks
named like files.*. I found that 100-line chunks works well;
it takes our system about a second on average to verify a chunk
- for f in files.*; do
p4 -x "$f" verify -q
sleep 4
done

The actual script has bells and whistles, like choosing a different
chunk size, sleep time, options for "p4 verify" (-u and -v).

In particular, doing "p4 verify -u" first can be useful periodically,
to compute and save checksums when they're missing.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

BTW, someone else said that in "p4 verify //path#1,#1", "#1,#1" could
be replaced by just "#1". The example in "p4 help verify" for
Perforce 2007.3 has

p4 verify -qu //...
p4 verify -q #1,#1
p4 verify -q #head,#head

[s***@vance.com]

For normal verifies, you are absolutely correct that it can swamp the
server temporarily. In his case, he's in a server-down situation with data
corruption. Contention with users is not an issue.

Steve

Original Message:
-----------------
From: Tim McDaniel ***@panix.com
Date: Thu, 1 Oct 2009 14:44:35 -0500 (CDT)
To: perforce-***@perforce.com
Subject: Re: [p4] How do i find corrupt depot files

Post by Anders Kjærgaard Hansen
I would very much be able to query our p4 server to go through all
files tonight, and report all files which have problems with its
files.

Someone mentioned "p4 verify -q //...".

I didn't like that so much because it takes 2 hours or so on my
workplace's repository, and I believe that it locks against submits
the entire time.

I did a script to somewhat ameliorate that. I'd post it except
- my job claims ownership of all code done for the job
- it's straightforward to do if you know how to script in
UNIX-like systems (or Cygwin).

The outline:
- renice $$ to a lower priority
- p4 files | sed -e 's/#.*//' > files
- use the split command to split files into chunks
named like files.*. I found that 100-line chunks works well;
it takes our system about a second on average to verify a chunk
- for f in files.*; do
p4 -x "$f" verify -q
sleep 4
done

The actual script has bells and whistles, like choosing a different
chunk size, sleep time, options for "p4 verify" (-u and -v).

In particular, doing "p4 verify -u" first can be useful periodically,
to compute and save checksums when they're missing.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

BTW, someone else said that in "p4 verify //path#1,#1", "#1,#1" could
be replaced by just "#1". The example in "p4 help verify" for
Perforce 2007.3 has

p4 verify -qu //...
p4 verify -q #1,#1
p4 verify -q #head,#head

[Anders Kjærgaard Hansen]

I totally agree Steve.

Sorry for the delay on getting back to this. Had an extended weekend planned, and as the server was up and running Thursday I kept to that good long weekend.

Just for the record, this is what we did.

First of all, the P4 server was shut down, as Vance stated it, with data corruption and a bad file system expected, nothing else seemed right.

We then copied all files in depot, db.* files, checkpoint and journal to a new disk.

On this disk I restored our license file, and started the server again on a different port, to be sure no users interfering in my recovery.

I then performed p4 verify stepwise for all files in the depot. None flagged as erroneous with MISSING or BAD.

I then did a compare between files in our depot and the backup from the night before the server upgrade - expecting this to be correct.

About a 1000 files flagged out as changed during these days, and I did some random manual verifying of these.

Being happy with the result, I started the server again, verified all seemed right, left the responsibility with my colleague, and went of for a nice long weekend.

And when I look in the logs for the backups since all seems right. Every Saturday we run a full verify on the entire depot, and all did verify.

Thanks for your support, regaining trust in the verify command saved me a lot of downtime.

Best regards
/Anders





-----Original Message-----
From: perforce-user-***@perforce.com [mailto:perforce-user-***@perforce.com] On Behalf Of ***@vance.com
Sent: Thursday, October 01, 2009 11:56 PM
To: ***@panix.com; perforce-***@perforce.com
Subject: Re: [p4] How do i find corrupt depot files

For normal verifies, you are absolutely correct that it can swamp the
server temporarily. In his case, he's in a server-down situation with data
corruption. Contention with users is not an issue.

Steve

Original Message:
-----------------
From: Tim McDaniel ***@panix.com
Date: Thu, 1 Oct 2009 14:44:35 -0500 (CDT)
To: perforce-***@perforce.com
Subject: Re: [p4] How do i find corrupt depot files

Post by Anders Kjærgaard Hansen
I would very much be able to query our p4 server to go through all
files tonight, and report all files which have problems with its
files.

Someone mentioned "p4 verify -q //...".

I didn't like that so much because it takes 2 hours or so on my
workplace's repository, and I believe that it locks against submits
the entire time.

I did a script to somewhat ameliorate that. I'd post it except
- my job claims ownership of all code done for the job
- it's straightforward to do if you know how to script in
UNIX-like systems (or Cygwin).

The outline:
- renice $$ to a lower priority
- p4 files | sed -e 's/#.*//' > files
- use the split command to split files into chunks
named like files.*. I found that 100-line chunks works well;
it takes our system about a second on average to verify a chunk
- for f in files.*; do
p4 -x "$f" verify -q
sleep 4
done

The actual script has bells and whistles, like choosing a different
chunk size, sleep time, options for "p4 verify" (-u and -v).

In particular, doing "p4 verify -u" first can be useful periodically,
to compute and save checksums when they're missing.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

BTW, someone else said that in "p4 verify //path#1,#1", "#1,#1" could
be replaced by just "#1". The example in "p4 help verify" for
Perforce 2007.3 has

p4 verify -qu //...
p4 verify -q #1,#1
p4 verify -q #head,#head
--
Tim McDaniel, ***@panix.com

--------------------------------------------------------------------
mail2web - Check your email from the web at
http://link.mail2web.com/mail2web

_______________________________________________
perforce-user mailing list - perforce-***@perforce.com
http://maillist.perforce.com/mailman/listinfo/perforce-user

[Jeff A. Bowles]

I think that I would rebuild the db* files from a clean checkpoint sometime
fairly soon. (As in hours or days, but no more than a week from now.)
---

Post by Anders Kjærgaard Hansen
I totally agree Steve.
Sorry for the delay on getting back to this. Had an extended weekend
planned, and as the server was up and running Thursday I kept to that good
long weekend.
Just for the record, this is what we did.
First of all, the P4 server was shut down, as Vance stated it, with data
corruption and a bad file system expected, nothing else seemed right.
We then copied all files in depot, db.* files, checkpoint and journal to a new disk.
On this disk I restored our license file, and started the server again on a
different port, to be sure no users interfering in my recovery.
I then performed p4 verify stepwise for all files in the depot. None
flagged as erroneous with MISSING or BAD.
I then did a compare between files in our depot and the backup from the
night before the server upgrade - expecting this to be correct.
About a 1000 files flagged out as changed during these days, and I did some
random manual verifying of these.
Being happy with the result, I started the server again, verified all
seemed right, left the responsibility with my colleague, and went of for a
nice long weekend.
And when I look in the logs for the backups since all seems right. Every
Saturday we run a full verify on the entire depot, and all did verify.
Thanks for your support, regaining trust in the verify command saved me a lot of downtime.
Best regards
/Anders
-----Original Message-----
Sent: Thursday, October 01, 2009 11:56 PM
Subject: Re: [p4] How do i find corrupt depot files
For normal verifies, you are absolutely correct that it can swamp the
server temporarily. In his case, he's in a server-down situation with data
corruption. Contention with users is not an issue.
Steve
-----------------
Date: Thu, 1 Oct 2009 14:44:35 -0500 (CDT)
Subject: Re: [p4] How do i find corrupt depot files

Post by Anders Kjærgaard Hansen
I would very much be able to query our p4 server to go through all
files tonight, and report all files which have problems with its
files.

Someone mentioned "p4 verify -q //...".
I didn't like that so much because it takes 2 hours or so on my
workplace's repository, and I believe that it locks against submits
the entire time.
I did a script to somewhat ameliorate that. I'd post it except
- my job claims ownership of all code done for the job
- it's straightforward to do if you know how to script in
UNIX-like systems (or Cygwin).
- renice $$ to a lower priority
- p4 files | sed -e 's/#.*//' > files
- use the split command to split files into chunks
named like files.*. I found that 100-line chunks works well;
it takes our system about a second on average to verify a chunk
- for f in files.*; do
p4 -x "$f" verify -q
sleep 4
done
The actual script has bells and whistles, like choosing a different
chunk size, sleep time, options for "p4 verify" (-u and -v).
In particular, doing "p4 verify -u" first can be useful periodically,
to compute and save checksums when they're missing.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
BTW, someone else said that in "p4 verify //path#1,#1", "#1,#1" could
be replaced by just "#1". The example in "p4 help verify" for
Perforce 2007.3 has
p4 verify -qu //...
p4 verify -q #1,#1
p4 verify -q #head,#head
--
--------------------------------------------------------------------
mail2web - Check your email from the web at
http://link.mail2web.com/mail2web
_______________________________________________
http://maillist.perforce.com/mailman/listinfo/perforce-user
_______________________________________________
http://maillist.perforce.com/mailman/listinfo/perforce-user

[Robert Cowham]

Post by Tim McDaniel
Someone mentioned "p4 verify -q //...".
I didn't like that so much because it takes 2 hours or so on
my workplace's repository, and I believe that it locks
against submits the entire time.

Verify doesn't lock the DB (so submits can happen while it is running) but
it does make the server do lots of work. Normally you wouldn't run it during
periods of load as your users will notice the server being rather slow as a
result.

If Anders is having file corruptions with license files being overwritten,
then I'd:

- consider the possibility my db.* had been corrupted and restore from
checkpoint/journals
- start with small verify's on only a few files, or the most recent 10 or
100 changelists before verifying everything
- be reviewing my backups like Steve said :)

Your script Tim is a good idea. I have done a variant with a Ruby tasks and
separate tasks so that verifies execute in parallel. This reduced the total
time for a very large repository from 48 hours to 16 hours for one client.
Sizeable numbers. Server pretty much unusable under such load!

Post by Tim McDaniel
- renice $$ to a lower priority
- p4 files | sed -e 's/#.*//' > files
- use the split command to split files into chunks
named like files.*. I found that 100-line chunks works well;
it takes our system about a second on average to verify a chunk
- for f in files.*; do
p4 -x "$f" verify -q
sleep 4
done
In particular, doing "p4 verify -u" first can be useful
periodically, to compute and save checksums when they're missing.

This is a hangover from pre-2003.2 or so clients/servers. It really oughtn't
to be required for more recent servers/clients (but doesn't hurt).

Robert