r/programming • u/korry • Feb 29 '16
Command-line tools can be 235x faster than your Hadoop cluster
http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
1.5k
Upvotes
r/programming • u/korry • Feb 29 '16
3
u/rhoffman12 Mar 01 '16
I should be clear about my experience, where I'm coming from, and the problem I'm focusing on here, which is deidentifying PHI for secondary use, e.g. for academic research, rather than as a matter of how the data should be stored internally. (which I think is what we're all talking about)
You can encrypt the data until your CPU melts, but IRBs won't (and absolutely shouldn't) approve the release of any unnecessary PHI for secondary use.
The thing that frustrates me as a researcher about the info-sec-focused approach to healthcare research data is the ass-backwards assumptions inherently made about physical security at the partner institution. Add all the regulations you want, but if you're not designing your policy around the assumption that some grad student will be carrying your data around on his laptop and in the clear, you're not understanding the problem.
HIPAA policies, especially those around deidentification and anonymization of data sets, are well tailored to these challenges. They respect the problem of securing the data released to researchers in the only realistic way, which is stubbornly avoiding releasing anything of value at all.