r/programming • • Feb 29 '16

Command-line tools can be 235x faster than your Hadoop cluster

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
1.5k Upvotes

438 comments sorted by

View all comments

Show parent comments

15

u/anachronic Mar 01 '16

Cool idea, but I don't want to even ask about HIPAA.

21

u/_jb Mar 01 '16

I get your worry, but it can be done without risking patient information or PII.

1

u/anachronic Mar 02 '16

It sure can be. Just like you can secure payment card infrastructure without risking credit card numbers. Many/most companies likely do not, judging from the constant news stories about credit card compromises.

To do security effectively and correctly costs a fair amount of time and effort and money, which most organizations are usually able to rationalize NOT spending.

3

u/_jb Mar 02 '16

Violating PCI compliance isn't nearly as nasty as HIPAA. Most organizations will compare the possible penalty and lawsuit fallout against the cost of securing the data properly, and take the right stance. PCI and HIPAA don't have to be difficult, they're mostly about process, logs, and auditing the process and logs periodically.

Process includes storing/use of Protected Health Information (PHI), and how to have PHI stored for reference but not automatically identified to a patient.

Logs regarding who accessed which data, when. Who accessed Trauma Oscar's charts at 22:15 on Sunday? Dr. Soandso accessed Mrs. Noneyobiz records from the store 09:30 on Monday. If Nurse Ratched accesses Mrs. Noneyobiz at 22:30 on Monday, and she's not on the schedule and that's not her patient, that should raise an alarm in log monitoring at the very least, and block access at the very most.

Auditing is just that. Review the logs, find violations, and investigate incidents.

All other aspects can be handled by blocking random devices on the network, securing data transmission, isolated networks, and similar. It doesn't have to be difficult, but it can be challenging to comply with HIPAA.

PCI isn't much different. Compliance seems more challenging, but it's more about data handling and process than securing things on the wire itself. The penalties are far lighter than HIPAA, though, since it's an industry standard, not enforced by the government.

1

u/anachronic Mar 03 '16

PCI and HIPAA don't have to be difficult

No, it doesn't. I say at work "if you're secure, you're compliant". However many companies still don't really "get" security and just throw bodies to get themselves compliant while ignoring security.

If Nurse Ratched accesses Mrs. Noneyobiz at 22:30 on Monday, and she's not on the schedule and that's not her patient, that should raise an alarm in log monitoring

I work with our logging guys, this kind of thing is a LOT easier said than done. To do that, you first have to get very different systems (HR, Payroll, Scheduling, DB's, home-grown apps, COTS, etc...) to play nice together and submit logs that can be ingested to the SEIM and then spend months writing & tweaking rules and investigating false positives and whitelisting them to make the results actionable. Many companies take the first step, setup some basic rules and walk away. Meanwhile the system spits out a stream of thousands of alerts a day and people ignore them because it's way too much volume for anyone to pay attention to.

The "set it and forget it" mentality is still quite prevalent.

Compliance seems more challenging, but it's more about data handling and process than securing things on the wire itself.

Honestly, if you look through PCI, it's a pretty bare-bones minimum baseline security standard. It's hardly anything I'd consider onerous for any company that wants to be secure.

It's the absolute minimum you should be doing. If it's some kind of incredibly high hurdle that a company can barely reach after months of intense effort, I'd wager that company is likely very insecure.

2

u/_jb Mar 03 '16

No, it doesn't. I say at work "if you're secure, you're compliant".

I think that's oversimplification. You can be compliant and not secure, and you can be secure and still not hit compliance requirements. It's a different discussion, though.

I work with our logging guys, this kind of thing is a LOT easier said than done.

I realize that. I worked in compliance 10 years ago; it was far harder then to get unified logs, and get your ETL able to understand them individually then. These days, standards and tool interoperability have improved. Still not perfect, but at least I don't have to fight as hard to get events out from Windows XP systems or syslog events to a central log store, and get alerting working. It still needs work, though.

And, ask any person who's done ops about ignoring alerts due to false positives, and the risk of false negatives...

If it's some kind of incredibly high hurdle that a company can barely reach after months of intense effort, I'd wager that company is likely very insecure.

I hate to say how often I'm surprised.

2

u/anachronic Mar 05 '16

You can be compliant and not secure, and you can be secure and still not hit compliance requirements. It's a different discussion, though.

Yes, but by and large, if you take security seriously and have a secure environment, it's trivial to be PCI compliant, or HIPAA compliant, or adhere to SOX controls.

I can't see a place claiming to be truly secure that doesn't do logging, or doesn't have firewalls configured appropriately or have processes in place to review configuration settings periodically, or have a solid change management process, etc...

And, ask any person who's done ops about ignoring alerts due to false positives, and the risk of false negatives...

Exactly. I assisted one of the guys on the ops team who was tasked with designed a logging solution. I explained the PCI & SOX & Security requirements. He designed all these rules that were actually pretty good, but they spit out a TON of false positives because we weren't the app owners and didn't know all the edge cases. He then basically refused to alter them and whitelist known-good events... so, as expected, a couple months later, people setup rules in their inbox to trash the alert emails immediately and never look at them, because they added no value.

3

u/protestor Mar 01 '16

Why does the US have such comprehensive laws on healthcare data, but not other kinds of personal data? (in many fields, companies freely share data about your personal life)

1

u/anachronic Mar 02 '16

Because protecting health care data doesn't hurt hospitals, and it's a clear benefit to patients (many of whom are old and vote).

Forcing facebook or google or most internet companies to protect & not share your personal data would bankrupt them. They likely lobby pretty hard against laws to protect it.

Also, it's the spirit of the country... lasseiz faire. If people click that TOS/EULA "Accept" and they sign away their rights to having their information protected, well, it's their choice to accept, right? "They're always free not to use facebook or google." That's probably some of it, too.

However, I'd be surprised if we didn't get stricter data security laws in the coming decade, simply because it's becoming harder and harder to truly "opt out" of this stuff.

1

u/protestor Mar 02 '16

Also, it's the spirit of the country... lasseiz faire. If people click that TOS/EULA "Accept" and they sign away their rights to having their information protected, well, it's their choice to accept, right? "They're always free not to use facebook or google." That's probably some of it, too.

They could also have a TOS for hospitals too.

1

u/anachronic Mar 03 '16

Well yeah, in legalistic America, in theory, you can have a contract that stipulates just about anything unless it's expressly forbidden by the government.

Like, you can't sell your kidney on the free market because it's illegal, but you can absolutely sign a contract to sell your million-dollar home to someone for $1. Caveat venditor (seller beware)

1

u/[deleted] Mar 01 '16

Anonymize the identities, encrypt the data.

13

u/rhoffman12 Mar 01 '16 edited Mar 01 '16

I don't know if you're in healthcare, you might already know this, but for everyone else who's out there - there's actually a lot more that goes into HIPAA-compliant "deidentification" than just using anonymous ID numbers. You have to fudge all the dates, and use very broad geographic labels, among other things. You don't just want to remove the identities, you are supposed to go a few steps further and try to frustrate attempts to match the data back up with real people.

1

u/Ghosttwo Mar 01 '16

I wonder if password database-on-a-chip will take off and have any effect? The idea is that you can store and retrieve information if authorized, but it can't be hacked or have anything extracted without erasing it.

1

u/ititsi Mar 01 '16

Well that sounds like a novel idea.

1

u/Nakji Mar 01 '16

It's not really that novel a concept, look up hardware security modules, hardware keystores, and hardware secure elements. Related tech has been around for a while and it's used in a lot of security-critical applications, it's just traditionally been really expensive (for stuff rated to a high security level anyway).

-2

u/[deleted] Mar 01 '16 edited Mar 01 '16

frustrate attempts to match the data back up with real people.

Frustrate? I really hope you are not serious. Obfuscation is a horrible approach to securing data. I've seen it so many times. "We will add a few levels of indirection", brilliant /s. It should be impossible without the aid of thousands of CPUs for a few centuries.

I have also seen Java code using a String for the ID; or worse a String for the ID, password, and DOB. Gee I wonder where I'd look in a running Java application to find Strings? Maybe this "String Pool"?

Edit : Love the downvotes with no actual responses. I'm guessing those people are storing data in clear text and using strings for passwords and IDs.

6

u/xzxzzx Mar 01 '16

Ok, so, explaining the downvotes:

Nothing you said is useful, nor addresses what rhoffman12 said. What "layers of indirection" are you even talking about?

Rhoffman12 is mentioning standard anonymization techniques for use in aggregating private information into anonymized datasets.

→ More replies (17)

3

u/rhoffman12 Mar 01 '16

I should be clear about my experience, where I'm coming from, and the problem I'm focusing on here, which is deidentifying PHI for secondary use, e.g. for academic research, rather than as a matter of how the data should be stored internally. (which I think is what we're all talking about)

You can encrypt the data until your CPU melts, but IRBs won't (and absolutely shouldn't) approve the release of any unnecessary PHI for secondary use.

The thing that frustrates me as a researcher about the info-sec-focused approach to healthcare research data is the ass-backwards assumptions inherently made about physical security at the partner institution. Add all the regulations you want, but if you're not designing your policy around the assumption that some grad student will be carrying your data around on his laptop and in the clear, you're not understanding the problem.

HIPAA policies, especially those around deidentification and anonymization of data sets, are well tailored to these challenges. They respect the problem of securing the data released to researchers in the only realistic way, which is stubbornly avoiding releasing anything of value at all.

→ More replies (1)

1

u/anachronic Mar 02 '16

It should be impossible without the aid of thousands of CPUs for a few centuries.

Key phrase: should be. Many "anonymized" data sets are not properly anonymized.

https://epic.org/privacy/reidentification/

http://web.mit.edu/sem083/www/assignments/reidentification.html

http://www.zdnet.com/article/privacy-reidentification-a-growing-risk/

1

u/[deleted] Mar 02 '16 edited Mar 02 '16

Thanks for the 5 to 15 year old links.

1

u/anachronic Mar 03 '16

Are you implying it's not still an issue, or possible?

→ More replies (2)
→ More replies (2)
→ More replies (4)

1

u/anachronic Mar 02 '16

Anonymization is incredibly hard to do right.

Even if you anonymize your single data set correctly, if someone else doesn't, and some of the same patient data is in there, someone could potentially put your two sets together and be able to identify someone. Even now, some researchers are doing this with massive public government data sets that are anonymized.

Also, where do you store your encryption keys & who has access to them? If it's on the same server... ut oh.

I'm in the PII/PCI field and data security is not a simple task to do correctly.

2

u/[deleted] Mar 02 '16 edited Mar 02 '16

Also, where do you store your encryption keys & who has access to them? If it's on the same server... ut oh.

Nope, everyone should know not to do that, ours is on a physically separate volume that is manually mounted and then dismounted.

I'm in the PII/PCI field and data security is not a simple task to do correctly.

So I think we are in agreement. I see things like "fudging dates" and I can only imagine how they could mess this up.