Closed Bug 1439019 Opened 6 years ago Closed 6 years ago

write script to compare data in two environments

Categories

(Socorro :: General, task, P1)

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: willkg, Assigned: willkg)

References

Details

We're building a new infrastructure and Socorro is very complex with non-trivial and very stateful data flows. One of the things we ask ourselves is, "Is the new infrastructure producing the same data as the old one?"

This question is hard to ask specifically since there's a lot of data involved and it has shapes and timeliness factors. It's also really hard to answer for exactly the same reasons.

This bug covers writing a script that uses existing APIs in the webapp to compare data generated by two environments and show where they differ.
Making this a P1. I mentioned this to Miles and he said he'd like something like this yesterday. This is a pretty critical time to have something like this and it's probably the case that anything we put together will help.

Grabbing this to work on next week.
Assignee: nobody → willkg
Status: NEW → ASSIGNED
Priority: -- → P1
Oops--blocked the wrong tracker. SO MANY TRACKERS!
Blocks: 1391034
No longer blocks: 1361394
I'm (ab)using pytest's diff things and wrote it as pytest tests:

https://github.com/willkg/socorro-compare

That has some tests done. It doesn't have tests for raw and processed crashes which I think we want. It probably needs to be less strict, too. But it's a good start.
I finished a first pass on all the things I was thinking we should compare:

* adi -- tests build_adu data flow
* product versions -- tests ftpscraper based data flow
* supersearch -- tests that the processor is processing the same crashes between two environments
* raw_crash -- tests that raw crashes look the same
* processed_crash -- tests that processed crashes look the same which also tests mdsw configuration and a bunch of other things

I think it's good enough for now. We can hone it going forward.

Marking as FIXED.
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.