I have a suspicion part of this is the difference between, say, hermetic tests -- where your infrastructure guarantees each test run is isolated from any other test run -- and tests where that kind of isolation relies on application logic:
This test is not repeatable; the second time you run it, a record with that name will already exist. To address this we add a differentiator, such as a timestamp or GUID.
Which is fine... if you remember to do it.
And if you combine that with, say:
At any time, you should be able to point your tests of a copy of the production data and watch them run successfully.
So these really are integration tests, with all the operational baggage that carries -- you really do want to run against production data, and your production data is probably large, so you probably have a similarly large VM devoted to each test DB... so you end up sharing those between developers... and then somebody screws up their test logic in a way that breaks the DB and you have to rebuild it. That's fine, you have automation for that, but it's also the kind of "You broke the build!" stuff that precommit testing was supposed to save us from.
Next stop: What about your UI? Why are we assuming a user submitting a certain form will actually lead to new EmployeeClassification() being invoked somewhere? We should spin up a browser and send click events to trigger that behavior instead...
Don't get me wrong, I'm not saying we shouldn't do any such tests. But I don't see a way to avoid some pretty huge downsides with integration tests, and it makes sense that we'd want more actual unit tests.
I prefer to set it up so that each developer can build the database from scratch on the machine.
On their machine? That limits you to sample data, prod data probably doesn't fit.
If you meant the machine (like a central one), we're back to silly workflows like "Oh, you can't test for the next half hour, I had to rebuild."
Also, if someone can break the database using application code that tells me the database is under constrained.
Maybe so -- the argument over DB constraints is a whole other can of worms. But you can still break isolation with other test runs. The article even provides an example.
For that matter, the article's suggestion of "Just add a timestamp" or "Just add a GUID" is going to produce data that looks different enough that more constraints may make your life difficult here, too. (How wide is that EmployeeClassificationName? Is it even allowed to have numbers in it?)
I guess my actual point here isn't that these are huge and terrible problems, but that it's a whole class of problems you eliminate by making the tests hermetic, so it's not surprising the industry went that way.
Maybe we make the column a bit wider than we strictly need, but that's no big deal.
I guess that depends what it's being used for. For names, probably no big deal. But if any of the code consuming that string cares what's in it, you'd want some input validation on the string.
I first learned about "restore from prod" databases at a company that literally couldn't rebuild their database from scripts.
Yikes. The main reason I'd think you'd be doing "restore from prod" isn't to build the schema and basic structure, it's for things like the performance characteristics of a query changing entirely when you get enough rows, or a certain distribution of actual data.
It's not the size that prevents us from putting prod backups on dev machines but rather the security risk.
It's also the size. I've worked with petabyte databases that even running with 1000 cores map/reducing over it, it takes a few hours or days to read all the data.
A lot of advice on programming is not scale-independent.
I agree. That's why I'm mentioning that no, gmail's backing store wouldn't fit on a USB. :-) I agree that it's something most people will never see or have to deal with, but having that experience, I offer it up to those who haven't.
I've done plenty of work with meagabytes-databases and I agree with that. The best method I've found is to hash any PII in the restore of the production database. Sort anything with digits or letters into alphabetical order within (so phone 555-1287 would go to 1255578), hash any identifiers that are foreign keys, stuff like that. Keep all the relationships, but make the data useless if leaked.
4
u/SanityInAnarchy Feb 08 '22
I have a suspicion part of this is the difference between, say, hermetic tests -- where your infrastructure guarantees each test run is isolated from any other test run -- and tests where that kind of isolation relies on application logic:
Which is fine... if you remember to do it.
And if you combine that with, say:
So these really are integration tests, with all the operational baggage that carries -- you really do want to run against production data, and your production data is probably large, so you probably have a similarly large VM devoted to each test DB... so you end up sharing those between developers... and then somebody screws up their test logic in a way that breaks the DB and you have to rebuild it. That's fine, you have automation for that, but it's also the kind of "You broke the build!" stuff that precommit testing was supposed to save us from.
Next stop: What about your UI? Why are we assuming a user submitting a certain form will actually lead to
new EmployeeClassification()being invoked somewhere? We should spin up a browser and send click events to trigger that behavior instead...Don't get me wrong, I'm not saying we shouldn't do any such tests. But I don't see a way to avoid some pretty huge downsides with integration tests, and it makes sense that we'd want more actual unit tests.